- Users - oVirt List Archives

vm paused unknown storage error one node out of 3 only
by David Gossage 11 Aug '16

11 Aug '16

Out of a 3 node cluster running oVirt 3.6.6.2-1.el7.centos with a 3 replicate gluster 3.7.14 starting a VM i just copied in on one node of the 3 gets the following errors. The other 2 the vm starts fine. All ovirt and gluster are centos 7 based. VM on start of the one node it tries to default to on its own accord immediately puts into paused for unknown reason. Telling it to start on different node starts ok. node with issue already has 5 VMs running fine on it same gluster storage plus the hosted engine on different volume. gluster nodes logs did not have any errors for volume nodes own gluster logs had this in log dfb8777a-7e8c-40ff-8faa-252beabba5f8 couldnt find in .glusterfs .shard or images/ 7919f4a0-125c-4b11-b5c9-fb50cc195c43 is the gfid of the bootable drive of the vm [2016-08-11 04:31:39.982952] W [MSGID: 114031] [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-2: remote operation failed [No such file or directory] [2016-08-11 04:31:39.983683] W [MSGID: 114031] [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-2: remote operation failed [No such file or directory] [2016-08-11 04:31:39.984182] W [MSGID: 114031] [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-0: remote operation failed [No such file or directory] [2016-08-11 04:31:39.984221] W [MSGID: 114031] [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-1: remote operation failed [No such file or directory] [2016-08-11 04:31:39.985941] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_txn] 0-GLUSTER1-replicate-0: Unreadable subvolume -1 found with event generation 3 for gfid dfb8777a-7e8c-40ff-8faa-252beabba5f8. (Possible split-brain) [2016-08-11 04:31:39.986633] W [MSGID: 114031] [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-2: remote operation failed [No such file or directory] [2016-08-11 04:31:39.987644] E [MSGID: 109040] [dht-helper.c:1190:dht_migration_complete_check_task] 0-GLUSTER1-dht: (null): failed to lookup the file on GLUSTER1-dht [Stale file handle] [2016-08-11 04:31:39.987751] W [fuse-bridge.c:2227:fuse_readv_cbk] 0-glusterfs-fuse: 15152930: READ => -1 gfid=7919f4a0-125c-4b11-b5c9-fb50cc195c43 fd=0x7f00a80bdb64 (Stale file handle) [2016-08-11 04:31:39.986567] W [MSGID: 114031] [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-0: remote operation failed [No such file or directory] [2016-08-11 04:31:39.986567] W [MSGID: 114031] [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-1: remote operation failed [No such file or directory] [2016-08-11 04:35:21.210145] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_txn] 0-GLUSTER1-replicate-0: Unreadable subvolume -1 found with event generation 3 for gfid dfb8777a-7e8c-40ff-8faa-252beabba5f8. (Possible split-brain) [2016-08-11 04:35:21.210873] W [MSGID: 114031] [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-1: remote operation failed [No such file or directory] [2016-08-11 04:35:21.210888] W [MSGID: 114031] [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-2: remote operation failed [No such file or directory] [2016-08-11 04:35:21.210947] W [MSGID: 114031] [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-0: remote operation failed [No such file or directory] [2016-08-11 04:35:21.213270] E [MSGID: 109040] [dht-helper.c:1190:dht_migration_complete_check_task] 0-GLUSTER1-dht: (null): failed to lookup the file on GLUSTER1-dht [Stale file handle] [2016-08-11 04:35:21.213345] W [fuse-bridge.c:2227:fuse_readv_cbk] 0-glusterfs-fuse: 15156910: READ => -1 gfid=7919f4a0-125c-4b11-b5c9-fb50cc195c43 fd=0x7f00a80bf6d0 (Stale file handle) [2016-08-11 04:35:21.211516] W [MSGID: 108008] [afr-read-txn.c:244:afr_read_txn] 0-GLUSTER1-replicate-0: Unreadable subvolume -1 found with event generation 3 for gfid dfb8777a-7e8c-40ff-8faa-252beabba5f8. (Possible split-brain) [2016-08-11 04:35:21.212013] W [MSGID: 114031] [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-0: remote operation failed [No such file or directory] [2016-08-11 04:35:21.212081] W [MSGID: 114031] [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-1: remote operation failed [No such file or directory] [2016-08-11 04:35:21.212121] W [MSGID: 114031] [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-2: remote operation failed [No such file or directory] I attached vdsm.log starting from when I spun up vm on offending node

1 0

Unable to login to the WEB UI
by Fabrice Bacchella 10 Aug '16

10 Aug '16

After the upgrad, I'm unable to log in, I'm getting the following error: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Where should I look to correct that ?

3 7

[ANN] oVirt 4.0.2 Fourth Release Candidate is now available
by Sandro Bonazzola 10 Aug '16

10 Aug '16

The oVirt Project is pleased to announce the availability of the Fourth Release Candidate of oVirt 4.0.2 for testing, as of August 10th, 2016. This release is available now for: * Fedora 23 (tech preview) * Red Hat Enterprise Linux 7.2 or later * CentOS Linux (or similar) 7.2 or later This release supports Hypervisor Hosts running: * Red Hat Enterprise Linux 7.2 or later * CentOS Linux (or similar) 7.2 or later * Fedora 23 (tech preview) * oVirt Next Generation Node 4.0 This is pre-release software. Please take a look at our community page[1] to know how to ask questions and interact with developers and users. All issues or bugs should be reported via oVirt Bugzilla[2]. This pre-release should not to be used in production. This update is the fourth release candidate of the second in a series of stabilization updates to the 4.0 series. 4.0.2 brings 24 enhancements and more than 200 bugfixes, including 107 high or urgent severity fixes, on top of oVirt 4.0 series See the release notes [3] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: * A new oVirt Live ISO is available. [4] * A new oVirt Next Generation Node will be available soon [4]. * A new oVirt Engine Appliance is already available. * Mirrors[5] might need up to one day to synchronize. Additional Resources: * Read more about the oVirt 4.0.2 release highlights: http://www.ovirt.org/release/4.0.2/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] https://www.ovirt.org/community/ [2] https://bugzilla.redhat.com/enter_bug.cgi?classification=oVirt [3] http://www.ovirt.org/release/4.0.2/ [4] http://resources.ovirt.org/pub/ovirt-4.0-pre/iso/ [5] http://www.ovirt.org/Repository_mirrors#Current_mirrors -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

1 0

engine-backup and messages in web admin events pane
by Gianluca Cecchi 10 Aug '16

10 Aug '16

Hello, having oVirt Engine Version: 4.0.0.6-1.el7.centos, when I run on engine VM a command like # engine-backup --mode=backup --scope=all --file=/root/engine-backup.archive_$(date +%d-%m-%Y) --log=/root/engine-backup_$(date +%d-%m-%Y).log Backing up: Notifying engine - Files - Engine database 'engine' - DWH database 'ovirt_engine_history' Packing into file '/root/engine-backup.archive_09-08-2016' Notifying engine Done. I see then that in engine web admin events pane I only have rows of kind: Finished Started without any reference to what... See here: https://drive.google.com/file/d/0BwoPbcrMv8mvYXNLZkZtQ1FFRUU/view?usp=shari… Is it me running backup in the wrong way or was it improved in 4.0.1? Thanks, Gianluca

2 2

ovirt-ha-agent and too many open files error
by Gianluca Cecchi 10 Aug '16

10 Aug '16

Hello, I have a 4.0 test environment (single host with self hosted engine) where I have 6 VMs defined (5 running) and no much activity. I do't monitor this system very much. Now I have connected to it to evaluate upgrade to 4.0.1 and see that about 15 days ago the ovirt-ha-agent died because of too many open files.... [root@ractor ovirt-hosted-engine-ha]# systemctl status ovirt-ha-agent -l ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring Agent Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; vendor preset: disabled) Active: inactive (dead) since Fri 2016-07-22 16:39:49 CEST; 2 weeks 4 days ago Main PID: 72795 (code=exited, status=0/SUCCESS) Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: self.set_file(fd) Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: File "/usr/lib64/python2.7/asyncore.py", line 657, in set_file Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: self.socket = file_wrapper(fd) Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: File "/usr/lib64/python2.7/asyncore.py", line 616, in __init__ Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: self.fd = os.dup(fd) Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: OSError: [Errno 24] Too many open files Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Shutting down the agent because of 3 failures in a row! Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Shutting down the agent because of 3 failures in a row! Jul 22 16:39:49 ractor.mydomain ovirt-ha-agent[72795]: WARNING:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:The VM is running locally or we have no data, keeping the domain monitor. Jul 22 16:39:49 ractor.mydomain ovirt-ha-agent[72795]: INFO:ovirt_hosted_engine_ha.agent.agent.Agent:Agent shutting down Is this sort of known problem or any reason to investigate? It seems very strange to have reached this limit I presume the agent runs as vdsm user and that the oVirt installation creates the file /etc/security/limits.d/99-vdsm.conf with # This limits are intended for medium VDSM hosts, for large hosts scale these # numbers appropriately. # nproc should be the maximum amount of storage operations usage. # VMs run by "qemu" user, vm processes are not relavent to "vdsm" user limits. vdsm - nproc 4096 # nofile should be at least 3(stdin,stdour,stderr) * each external process. # 3 * 4096 = 12288 vdsm - nofile 12288 As a rough estimation (over estimation actually , due to many duplicates) I have now: # lsof -u vdsm | wc -l 488 Anything else to check? Gianluca

2 2

Re: [ovirt-users] Migrate machines in unknown state?
by Nicolás 10 Aug '16

10 Aug '16

El 04/08/16 a las 15:25, Arik Hadas escribió: > > ----- Original Message ----- >> El 2016-08-04 08:24, Arik Hadas escribió: >>> ----- Original Message ----- >>>> >>>> El 04/08/16 a las 07:18, Arik Hadas escribió: >>>>> ----- Original Message ----- >>>>>> Hi, >>>>>> >>>>>> We're running oVirt 4.0.1 and today I found out that one of our hosts >>>>>> has all its VMs in an unknown state. I actually don't know how (and >>>>>> when) did this happen, but I'd like to restore service possibly without >>>>>> turning off these machines. The host is up, the VMs are up, 'qemu' >>>>>> process exists, no errors, it's just the VMs running on it that have a >>>>>> '?' where status is defined. >>>>>> >>>>>> Is it safe in this case to simply modify database and set those VM's >>>>>> status to 'up'? I remember having to do this a time ago when we faced >>>>>> storage issues, it didn't break anything back then. If not, is there a >>>>>> "safe" way to migrate those VMs to a different host and restart the >>>>>> host >>>>>> that marked them as unknown? >>>>> Hi Nicolás, >>>>> >>>>> I assume that the host these VMs are running on is empty in the >>>>> webadmin, >>>>> right? if that is the case then you've probably hit [1]. Changing their >>>>> status to up is not the way to go since these VMs will not be monitored. >>>> Hi Arik, >>>> >>>> By "empty" you mean the webadmin reports the host being running 0 VMs? >>>> If so, that's not the case, actually the VM count seems to be correct >>>> in >>>> relation to "qemu-*" processes (about 32 VMs), I can even see the >>>> machines in the "Virtual machines" tab of the host, it's just they are >>>> all marked with the '?' mark. >>> No, I meant the 'Host' column in the Virtual Machines tab but if you >>> see >>> the VMs in the "Virtual machines" sub-tab of the host then run_on_vds >>> points to the right host.. >>> >>> The host is up in the webadmin as well? >>> Can you share the engine log? >>> >> Yes, the host is up in the webadmin, there are no issues with it, just >> the VMs running on it have the '?' mark. I've made 3 tests: >> >> 1) Restart engine: did not help >> 2) Check firewall, seems to be ok. >> 2) PostgreSQL: UPDATE vm_dynamic SET status = 1 WHERE status = 8; : >> After a while, I see lots of entries like this: >> >> 2016-08-04 09:23:10,910 WARN >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (DefaultQuartzScheduler4) [6ad135b8] Correlation ID: null, Call Stack: >> null, Custom Event ID: -1, Message: VM xxx is not responding. >> >> I'm attaching the engine log, but I don't know when did this happen for >> the first time, though. If there's a manual way/command to migrate VMs >> to a different host I'd appreciate a hint about it. >> >> Is it safe to restart vdsmd on this host? > The engine log looks fine - the VMs are reported as not-responding for > some reason. I would restart libvirtd and vdsmd then Is restarting those two daemons safe? I mean, will that stop all qemu-* processes, so the VMs marked as unknown will stop? >> Thanks. >> >>>> Thanks. >>>> >>>>> Yes, there is no other way to resolve it other than changing the DB but >>>>> the change should be to update run_on_vds field of these VMs to the host >>>>> you know they are running on. Their status will then be updates in 15 >>>>> sec. >>>>> >>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1354494 >>>>> >>>>> Arik. >>>>> >>>>>> Thanks. >>>>>> >>>>>> Nicolás >>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>> >>>>

4 4

problem setting up bonding via rest in ovirt 3.6.6.2
by Sven Kieske 09 Aug '16

09 Aug '16

--j9oOhf6Grf0fFqlEVUxQMWAcfqJklkQI7 Content-Type: multipart/mixed; boundary="T7QE2T6V8U03TuXieVjo1CtXdRsCW5TA8" From: Sven Kieske <s.kieske(a)mittwald.de> To: users(a)ovirt.org Cc: devel <devel(a)ovirt.org> Message-ID: <69e3d02d-ac52-37d7-e8f6-5bc8be8c0d40(a)mittwald.de> X-Authenticated-mymxserver.com: Yes Subject: problem setting up bonding via rest in ovirt 3.6.6.2 --T7QE2T6V8U03TuXieVjo1CtXdRsCW5TA8 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, I'd like to setup host network in the following way: I have 4 interfaces: eth0, eno1, eno2, eno3 ovirtmgmt network resides on eth0 and is working. Now I want to setup a bonding with the slave interfaces eno2 and eno3 via the hosts "setupnetworks" function: URL: https://engine-fqdn/api/hosts/9cdcc85b-dd87-4dcf-9275-b993ee5b5e9c/nics/s= etupnetworks Request: { "modified_bonds": { "host_nic": [{ "name": "bond0", "bonding": { "options": { "option": [{ "name": "mode", "value": "1" }, { "name": "miimon", "value": "100" }] }, "slaves": { "host_nic": [{ "name": "eno2" }, { "name": "eno3" }] } } }] } } Response from ovirt-engine: <?xml version=3D"1.0" encoding=3D"UTF-8" standalone=3D"yes"?> <fault> <reason>Incomplete parameters</reason> <detail>Action [hostNics] required for setupNetworks</detail> </fault> When I look at the rsdl docs this seems perfectly fine. I tried google with this error condition but it didn't return anything useful unfortunately. Your help would be greatly appreciated. The official RH docs also do not mention anything else: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtuali= zation/3.6/html-single/REST_API_Guide/index.html#Multiple_Network_Setup_A= ction https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtuali= zation/3.6/html-single/REST_API_Guide/index.html#Bonded_Interfaces Also this tool doesn't seem to mention anything else: https://jhernand.fedorapeople.org/ovirt-api-explorer/#/services/host/meth= ods/setup_networks --=20 Mit freundlichen Gr=C3=BC=C3=9Fen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH & Co. KG K=C3=B6nigsberger Stra=C3=9Fe 6 32339 Espelkamp T: +495772 293100 F: +495772 293333 https://www.mittwald.de Gesch=C3=A4ftsf=C3=BChrer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhause= n Komplement=C3=A4rin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oey= nhausen --T7QE2T6V8U03TuXieVjo1CtXdRsCW5TA8-- --j9oOhf6Grf0fFqlEVUxQMWAcfqJklkQI7 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJXqfNTAAoJEMby9TMDAbQR9NwQAJ4xm0JeDqRnjW8DuAae6wCz RLWbGMyhn5jqcL7bRQAU/oppW0FAFGo+So72kwvX29AkII+qRZJnHyHWMo6nnO3s XnCR23uAk33KH02jmyzVNq9cA6VmckTHX3CWU6cq6WxTA8jhDKNSu7SLkUvOD3/u pUDGNvoYZJTsN2DU6qQJiROMkIu+zgm0LF9RXKdVcA+tE1G9XuOwz+QBqgAfiWgc bCD3mWIF+OBfwQoN1GIhe7sxZm3TxJs+Y7UtpotN/hYULXIJffsjU/Lo6e5pEYJY aPIfPUscPlAr51alEiR11zN9pgzhShqvRW61GHxsxkxTXPn8JVsch2uD4sCe8pQn 8C8Tz9+gdq0tM/gvM3nyYTIcHmMr7ZDn44lyd+x98uJWsUNOuraQH18nAIrs+kaw TfonDACJKM5t8DQM1FgmUA0nWdS3X5UCcY8LSIHpA5dhLk9sRUWehfvqPl1zDrpJ uTcOF5/POD7vgpf4Aja9vOlcgrKZJGrKPWGDq2IJkX1OoaeGe2KsIf3VqzqxFG0g l8UlR9zIeXwslLcfvRpt4YUJT27l3zHkaC6glZo+vkXbvb0fUPVwKfV3Xcyaz9G+ BIs1dC1iYrVGUURaDu8yZ/SZQ5zigD/I/wudKvTgelR8Wd8Y3MStmpkADozqBwY3 hQG0y+cMX/7gvpOsD7cw =0XKM -----END PGP SIGNATURE----- --j9oOhf6Grf0fFqlEVUxQMWAcfqJklkQI7--

2 1

3.6 : Hosted_Storage unattached
by Alexis HAUSER 09 Aug '16

09 Aug '16

Hi, I installed a new node with a new hosted engine, version 3.6, added a data domain, but I can't see the hosted_storage. I tried to use the "import storage" on it but it keeps having unattached status, and in the logs I can see : "2016-08-09 05:39:32,821 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [] Correlation ID: 1d9c8f67, Job ID: 6e75d61e-3140-4f1c-a301-e683ddc28b1f, Call Stack: null, Custom Event ID: -1, Message: This Data center compatibility version does not support importing a data domain with its entities (VMs and Templates). The imported domain will be imported without them." Any ideas ?

2 2

oVirt 4 and Gluster network
by Arsène Gschwind 09 Aug '16

09 Aug '16

Hi, We have setup an oVirt 4 cluster using Gluster as storage backend. I've setup a separate 10 Gb Network for Gluster traffic and tried to use that network as written in : https://www.ovirt.org/develop/release-management/features/network/select-ne… But I'm not able to find the "Edit - Brick" menu to be able to set the IP to use for existing bricks. Is this not implemented yet? The feature status says "done". I've been able to create new bricks using that network and it seems the traffic is going trough the right nics. Thanks for any hint. regards, Arsène

2 1

Hosted Engine Storage Domain Stuck and no Volumes tab?
by Dan Lavu 09 Aug '16

09 Aug '16

So this is on 4.0.1, Fedora 23, the engine is running fine on glusterfs, on a 3 replica arbiter 1 volume, the hosted_engine attached storage domain is *stuck* in offline, wondernig if there is anyway to fix that? Since the domain cannot be managed inside the engine. Also I have gluster enabled for one of my clusters but I'm missing a volumes tab to manage the gluster volumes? The hosts, when putting them into maintenaince, I have the option to turn off gluster... I know in 3.5-3.6 volume management was a feature, did this change? Thanks, Dan

2 2