Re: [Users] Failing migration, inconsistent state
by Martijn Grendelman
Martijn Grendelman schreef op 5-12-2013 11:00:
> Hi,
>
> I tried to migrate several VMs from one host to another. Two VMs
> migrated without issues, but for one VM, the migration didn't happen. It
> seems to be hanging, but the UI is now in an inconsistent state:
>
> - The 'Tasks' tab reports 0 active tasks, but the last task (the
> migration in question) is still reported as 'Executing'.
> - The VM status is 'Up' (not migrating)
> - 'Migrate' action is choosable from menu, while 'Cancel Migration' is
> greyed out, but when I choose 'Migrate' and pick a host, I am told
> 'Cannot migrate VM. VM <name> is being migrated.'.
>
> What is the best way to fix this?
And perhaps this information is useful to oVirt developers:
> Thread-600648::DEBUG::2013-12-03 12:37:47,926::vm::180::vm.Vm::(_setupVdsConnection) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Destination server is: onode0.isaac.local:54321
> Thread-600648::DEBUG::2013-12-03 12:37:47,927::vm::182::vm.Vm::(_setupVdsConnection) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Initiating connection with destination
> Thread-600648::DEBUG::2013-12-03 12:37:47,990::vm::232::vm.Vm::(_prepareGuest) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Migration started
> Thread-600648::DEBUG::2013-12-03 12:37:48,006::vm::299::vm.Vm::(run) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::migration semaphore acquired
> Thread-600648::DEBUG::2013-12-03 12:37:48,115::vm::357::vm.Vm::(_startUnderlyingMigration) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::starting migration to qemu+tls://onode0.isaac.local/system with miguri tcp://onode0.isaac.local
> Thread-600648::DEBUG::2013-12-03 12:43:10,819::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 9 edom: 10 level: 2 message: operation failed: migration job: unexpectedly failed
> Thread-600648::DEBUG::2013-12-03 12:43:10,819::vm::742::vm.Vm::(cancel) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::canceling migration downtime thread
> Thread-600648::DEBUG::2013-12-03 12:43:10,819::vm::812::vm.Vm::(stop) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::stopping migration monitor thread
> Thread-600648::ERROR::2013-12-03 12:43:10,820::vm::238::vm.Vm::(_recover) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::operation failed: migration job: unexpectedly failed
> Thread-600648::ERROR::2013-12-03 12:43:11,276::vm::321::vm.Vm::(run) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Failed to migrate
> Traceback (most recent call last):
> File "/usr/share/vdsm/vm.py", line 308, in run
> self._startUnderlyingMigration()
> File "/usr/share/vdsm/vm.py", line 385, in _startUnderlyingMigration
> None, maxBandwidth)
> File "/usr/share/vdsm/vm.py", line 835, in f
> ret = attr(*args, **kwargs)
> File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper
> ret = f(*args, **kwargs)
> File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1178, in migrateToURI2
> if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
> libvirtError: operation failed: migration job: unexpectedly failed
Cheers,
Martijn.
10 years, 4 months
[Users] Problem introducing VDSM hosts
by Garry Tiedemann
Hi folks,
We are setting up ovirt on Proliant BL460c blades (mix of G1 & G6), and
found an interesting problem.
Can anyone here can offer guidance on how to approach this please?
Also, I would be interested to know others' experiences of using ovirt
on BladeSystem gear.
The problem: When trying to introduce a new VDSM host, the install
fails, immediately.
It's reproducible; we tried more than a dozen times, with different
combinations of NICs, to eliminate hardware faults.
These messages appear on the VDSM host:
Dec 13 10:55:24 bl05 sshd[2627]: pam_unix(sshd:session): session opened
for user root by (uid=0)
Dec 13 10:55:24 bl05 sshd[2627]: Corrupted MAC on input.
Dec 13 10:55:24 bl05 sshd[2627]: Disconnecting: Packet corrupt
Dec 13 10:55:24 bl05 sshd[2627]: pam_unix(sshd:session): session closed
for user root
It seems to be a checksum problem.
Google has plenty of similar reports (none ovirt-related), and very
little in the way of troubleshooting tips.
We have eliminated physical hardware; memory has tested OK, and
alternative network ports have been tried, with the same result.
I expected to be able to reproduce the issue outside ovirt, by scping
large files back and forth, however, this did not cause errors.
I believe it's a software matter, possibly TOE, but I don't know how to
take it further.
We are using FC19 for both the ovirt-engine and vdsm hosts.
Any guidance on how to solve this would be much appreciated.
Thanks in advance,
Garry
10 years, 4 months
[Users] Keepalived on oVirt Hosts has engine networking issues
by Andrew Lau
Hi,
I have the scenario where I have gluster and ovirt hosts on the same box,
to keep the gluster volumes highly available incase a box drops I'm using
keepalived across the boxes and using that IP as the means for the storage
domain. I documented my setup here in case anyone needs a little more info
http://www.andrewklau.com/returning-to-glusterized-ovirt-3-3/
However, the engine seems to be picking up the floating IP assigned to
keepalived as the interface and messing with the ovirtmgmt migration
network, so migrations are failing as my floating IP gets assigned to the
ovirtmgmt bridge in the engine however it's not actually there on most
hosts (except one) so vdsm seems to report destination same as source.
I've since created a new vlan interface just for storage to avoid the
ovirtmgmt conflict, but the engine will still pick up the wrong IP on the
storage vlan because of keepalived. This means I can't use the save network
feature within the engine as it'll save the floating ip rather than the one
already there. Is this a bug or just the way it's designed.
eth0.2 -> ovirtmgmt (172.16.0.11) -> management and migration network ->
engine sees, sets and saves 172.16.0.11
eth0.3 -> storagenetwork (172.16.1.11) -> gluster network -> engine sees,
sets and saves 172.16.1.5 (my floating IP)
I hope this makes sense.
p.s. can anyone also confirm, does gluster support multi pathing by
default? If I'm using this keepalived method, am I bottle necking myself to
one host?
Thanks,
Andrew
10 years, 4 months
[Users] errors in engine log and tones of sync messages
by Juan Pablo Lorier
Hi,
I'm trying to debug why I can't import a VM to my brand new glusterfs
domain and I see that engine.log is full of gluster volumelist calls and
less but often, errors of not been able to call async, I'm I missing any
package to install in the engine other than vdsm-gluster?
Is there a way not to log all the VolumeList commands (without lowering
log level) to avoid having so many entries that difficult searching in
the log?
Regards,
: 5c8baf53
2013-12-13 13:27:58,504 ERROR
[org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl]
(DefaultQuartzScheduler_Worker-33) Failed to invoke scheduled method
gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
[:1.7.0_45]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_45]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_45]
at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60)
[scheduler.jar:]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
[quartz.jar:]
Caused by: org.ovirt.engine.core.common.errors.VdcBLLException:
VdcBLLException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method
"glusterTasksList" is not supported (Failed with error VDS_NETWORK_ERROR
and code 5022)
at
org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:122) [bll.jar:]
at
org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33)
[bll.jar:]
at
org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService.runVdsCommand(GlusterTasksService.java:60)
[bll.jar:]
at
org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService.getTaskListForCluster(GlusterTasksService.java:28)
[bll.jar:]
at
org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob.updateTasksInCluster(GlusterTasksSyncJob.java:67)
[bll.jar:]
at
org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob.updateGlusterAsyncTasks(GlusterTasksSyncJob.java:56)
[bll.jar:]
... 6 more
2013-12-13 13:28:00,655 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-68) START,
GlusterVolumesListVDSCommand(HostName = Dell4, HostId =
001f72ef-fac6-42bc-be8e-b17e249e80bd), log id: 75cbf869
2013-12-13 13:28:00,692 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-68) FINISH, GlusterVolumesListVDSCommand,
return:
{30759593-86cf-47d3-8527-d36623674ea6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@ac95f510},
log id: 75cbf869
2013-12-13 13:28:05,743 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-67) START,
GlusterVolumesListVDSCommand(HostName = Dell3, HostId =
342b037f-e4ab-4c3b-b804-806d3ec30ba9), log id: 76386ce
2013-12-13 13:28:07,241 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-67) FINISH, GlusterVolumesListVDSCommand,
return:
{30759593-86cf-47d3-8527-d36623674ea6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@ee3bee7d},
log id: 76386ce
10 years, 4 months
[Users] simple networking?
by Ted Miller
I am trying to set up a testing network using o-virt, but the networking is
refusing to cooperate. I am testing for possible use in two different
production setups.
My previous experience has been with VMWare. I have always set up a single
bridged network on each host. All my hosts, VMs, and non-VM computers were
peers on the LAN. They could all talk to each other, and things worked very
well. There was a firewall/gateway that provided access to the Internet, and
hosts, VMs, and could all communicate with the Internet as needed.
o-virt seems to be compartmentalizing things beyond all reason.
Is there any way to set up simple networking, so ALL computers can see each
other?
Is there anywhere that describes the philosophy behind the networking setup?
What reason is there that networks are so divided?
After banging my head against the wall trying to configure just one host, I
am very frustrated. I have spent several HOURS Googling for a coherent
explanation of how/why networking is supposed to work, but only fine obscure
references like "letting non-VMs see VM traffic would be a huge security
violation". I have no concept of what king of an installation the o-virt
designers have in mind, but it is obviously worlds different from what I am
trying to do.
The best I can tell, o-virt networking works like this (at least when you
have only one NIC):
there must be an ovirtmgt network, which cannot be combined with any other
network.
the ovirtmgt network cannot talk to VMs (unless that VM is running the
engine)
the ovirtmgt network can only talk to hosts, not to other non-VM computers
a VM network can talk only to VMs
cannot talk to hosts
cannot talk to non-VMs
hosts cannot talk to my LAN
hosts cannot talk to VMs
VMs cannot talk to my LAN
All of the above are enforced by a boatload of firewall rules that o-virt
puts into every host and VM under its jurisdiction.
All of the above is inferred from things I Googled, because I can't find
anywhere that explains what or how things are supposed to work--only things
telling people WHAT THEY CANT DO. All I see on the mailing lists is people
getting their hands slapped because they are trying to do SIMPLE SETUPS that
should work, but don't (due to either design restrictions or software bugs).
My use case A:
* My (2 or 3) hosts have only one physical NIC.
* My VMs exist to provide services to non-VM computers.
* The VMs do not run X-windows, but they provide GUI programs to
non-VMs via "ssh -X" connections.
* MY VMs need access to storage that is shared with hosts and non-VMs on
the LAN.
Is there some way to TURN OFF network control in o-virt? My systems are
small and static. I can hand-configure the networking a whole lot easier
than I can deal with o-virt (as I have used it so far). Mostly I would need
to be able to turn off the firewall rules on both hosts and VMs.
banging head against wall,
Ted
10 years, 4 months
[Users] Upgrade from 3.3.1-2 el6 to 3.3.2
by Sven Kieske
Hi,
got a question regarding engine upgrades:
My current setup:
I got 3.3.1-2.el6 installed from ovirt-beta repo
and want to upgrade to 3.3.2 RC.
According to:
http://www.ovirt.org/OVirt_3.3.2_release_notes
"oVirt 3.3.2 release candidate is available for testing. In order to
install it you've to enable ovirt-updates-testing repository."
I need ovirt-updates-testing enabled?
Shall I disable ovirt-beta repo?
Additional info: the base os is still CentOS 6.4
Did anyone perform an inplace upgrade to 6.5?
Is this recommended?
--
Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator
Mittwald CM Service GmbH & Co. KG
Königsberger Straße 6
32339 Espelkamp
T: +49-5772-293-100
F: +49-5772-293-333
https://www.mittwald.de
Geschäftsführer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
10 years, 4 months
[Users] Documentation incoming
by Sven Kieske
Hi,
just for all who are curious:
There is a large package of documentation for oVirt
at redhat.com:
https://access.redhat.com/site/documentation/Red_Hat_Enterprise_Virtualiz...
There is also beta documentation for 3.3.
This is more structured than the ovirt.org wiki.
Maybe this helps someone.
PS: In case you do not know:
Red Hat Enterprise Virtualization (RHEV for short)
is a downstream product build from oVirt, so
almost everything in the docs should be transferable
to oVirt.
--
Mit freundlichen Grüßen / Regards
Sven Kieske
Systemadministrator
Mittwald CM Service GmbH & Co. KG
Königsberger Straße 6
32339 Espelkamp
T: +49-5772-293-100
F: +49-5772-293-333
https://www.mittwald.de
Geschäftsführer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
10 years, 4 months
[Users] Authentication
by Maurice James
--_fd849e30-761c-499c-ae24-10dcc6356ed6_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
I was curious to know how is the update of the authentication options comin=
g. Right now the only option for external authentication involves the use o=
f Kerberos. I am interested in using an LDAP only model (389-ds) without Ke=
rberos. =
--_fd849e30-761c-499c-ae24-10dcc6356ed6_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<style><!--
.hmmessage P
{
margin:0px=3B
padding:0px
}
body.hmmessage
{
font-size: 12pt=3B
font-family:Calibri
}
--></style></head>
<body class=3D'hmmessage'><div dir=3D'ltr'>I was curious to know how is the=
update of the authentication options coming. Right now the only option for=
external authentication involves the use of Kerberos. I am interested in u=
sing an LDAP only model (389-ds) without Kerberos. </div></body>
</html>=
--_fd849e30-761c-499c-ae24-10dcc6356ed6_--
10 years, 4 months
[Users] hot add ram to a vm
by Nathanaël Blanchet
This is a multi-part message in MIME format.
--------------080308040608050209070102
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
Hello,
Is hot add ram to a vm is supported by ovirt, or is it planned to be?
--
Nathanaël Blanchet
Supervision réseau
Pôle exploitation et maintenance
Département des systèmes d'information
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5
Tél. 33 (0)4 67 54 84 55
Fax 33 (0)4 67 54 84 14
blanchet(a)abes.fr
--------------080308040608050209070102
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<font size="-1">Hello,<br>
<br>
Is hot add ram to a vm is supported by ovirt, or is it planned to
be?<br>
</font>
<pre class="moz-signature" cols="72">--
Nathanaël Blanchet
Supervision réseau
Pôle exploitation et maintenance
Département des systèmes d'information
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5
Tél. 33 (0)4 67 54 84 55
Fax 33 (0)4 67 54 84 14
<a class="moz-txt-link-abbreviated" href="mailto:blanchet@abes.fr">blanchet(a)abes.fr</a> </pre>
</body>
</html>
--------------080308040608050209070102--
10 years, 4 months
