- Users - oVirt List Archives

IO Storage Error / All findings / Need help.
by Christian Reiss 24 Feb '20

24 Feb '20

Hello folks, I am running a 3-way, no arbiter Gluster setup using oVirt and contained Gluster 6.7. After a crash we are unable to start any VMs due to Storage IO error. After much, much backtracking and debugging we are closing in on the symptons, albeit not the issue. Conditions: - gluster volume is healthy, - No outstanding heal or split-brain files, - 3 way without arbiter nodes (3 copies), - I already ran several "heal full" commands. Gluster Volume Info Volume Name: ssd_storage Type: Replicate Volume ID: d84ec99a-5db9-49c6-aab4-c7481a1dc57b Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: node01.company.com:/gluster_bricks/ssd_storage/ssd_storage Brick2: node02.company.com:/gluster_bricks/ssd_storage/ssd_storage Brick3: node03.company.com:/gluster_bricks/ssd_storage/ssd_storage Options Reconfigured: cluster.self-heal-daemon: enable cluster.granular-entry-heal: enable storage.owner-gid: 36 storage.owner-uid: 36 network.ping-timeout: 30 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off performance.strict-o-direct: on transport.address-family: inet nfs.disable: on performance.client-io-threads: on Gluster Volume Status Status of volume: ssd_storage Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick node01.company.com:/gluster_br icks/ssd_storage/ssd_storage 49152 0 Y 8218 Brick node02.company.com:/gluster_br icks/ssd_storage/ssd_storage 49152 0 Y 23595 Brick node03.company.com:/gluster_br icks/ssd_storage/ssd_storage 49152 0 Y 8080 Self-heal Daemon on localhost N/A N/A Y 66028 Self-heal Daemon on 10.100.200.12 N/A N/A Y 52087 Self-heal Daemon on node03.company.com et N/A N/A Y 8372 Task Status of Volume ssd_storage ------------------------------------------------------------------------------ There are no active volume tasks The mounted path where the oVirt vm files reside is 100% okay, we copied all the images out there onto standalone hosts and the images run just fine. There is no obvious data corruption. However launching any VM out of oVirt fails with "IO Storage Error". This is where everything gets funny. oVirt uses a vdsm user to access all the files. Findings: - root can read, edit and write all files inside the ovirt mounted gluster path. - vdsm user can write to new files regardless of size without any issues; changes get replicated instantly to other nodes. - vdsm user can append to existing files regardless of size without any issues; changes get replicated instantly to other nodes. - vdsm user can read files if those files are smaller than 64mb. - vdsm user gets permission denied errors if the file to be read is 65mb or bigger. - vdsm user gets permission denied errors if the requests crosses a gluster shard-file boundary. - if root does a "dd if=file_larger_than64mb" of=/dev/null" on any large file, the file can then be read by the vdsm user on that single node. Changes do not get replicated to other nodes. Example: id of the vdsm user & sudo to them: [vdsm@node01:/rhev/data-center/mnt/glusterSD/node01.company.com:_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/test] $ id uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [vdsm@node02:/rhev/data-center/mnt/glusterSD/node01.company.com:_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/test] $ id uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [vdsm@node03:/rhev/data-center/mnt/glusterSD/node01.company.com:_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/test] $ id uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 Create a file >64mb on one node: [vdsm@node03:/rhev/data-center/mnt/glusterSD/node01.company.com:_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/test] $ base64 /dev/urandom | head -c 200000000 > file.txt [vdsm@node03:/rhev/data-center/mnt/glusterSD/node01.company.com:_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/test] $ ls -lha total 191M drwxr-xr-x. 2 vdsm kvm 30 Feb 4 13:10 . drwxr-xr-x. 6 vdsm kvm 80 Jan 1 1970 .. -rw-r--r--. 1 vdsm kvm 191M Feb 4 13:10 file.txt File is instantly available on another node: [vdsm@node01:/rhev/data-center/mnt/glusterSD/node01.company.com:_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/test] $ ls -lha total 191M drwxr-xr-x. 2 vdsm kvm 30 Feb 4 13:10 . drwxr-xr-x. 6 vdsm kvm 80 Jan 1 1970 .. -rw-r--r--. 1 vdsm kvm 191M Feb 4 13:10 file.txt Accessing the whole file fails: [vdsm@node01:] dd if=file.txt of=/dev/null dd: error reading ‘file.txt’: Permission denied 131072+0 records in 131072+0 records out 67108864 bytes (67 MB) copied, 0.0651919 s, 1.0 GB/s Reading first 64mb works, 65mb (crossing boundary) does not: [vdsm@node01:] $ dd if=file.txt bs=1M count=64 of=/dev/null 64+0 records in 64+0 records out 67108864 bytes (67 MB) copied, 0.00801663 s, 8.4 GB/s [vdsm@node01:] $ dd if=file.txt bs=1M count=65 of=/dev/null dd: error reading ‘file.txt’: Permission denied 64+0 records in 64+0 records out 67108864 bytes (67 MB) copied, 0.00908712 s, 7.4 GB/s Attaching/ appending to the file works (not crossing bounary): [vdsm@node01:] $ date >> file.txt [vdsm@node01:] $ [vdsm@node02:] $ tail -n2 file.txt E16ACZaLqLhx2oUUUov5JHvQcVFohn6HH+eog6XZCiTaG0Tue 4 Feb 13:18:37 CET 2020 Reading the file beginning & end works, if it crosses the boundary not so much: [vdsm@node02:] $ head file.txt jrZOxGaGvwfpGSwn1BKWWmFC4556KNzXsD2BCwY78tnV1mRY54IxnE+hbnszRyWgVuXhBpVRoJTp xvVwktZwSytMyvJjsSt7pQbXbHSY66tRe/rvrw5dHr3RNJn9HjqtlKQ9mHVX4ch1HkU5posSmDbg vwzxBTXWfxLDMmIghyTgBTSFiI9Xg8W6htxDpxrbO+10EzlnaN1Am5tAlTkfrorNLyihpiQhUPGG ag6tJUcFj3IySGRTAxnStFRQoBXN5dlyx1Sqc4s/Tpl7gkgR8+I7UcdRKISjgcGcpW+zrXKqFF/H Dwv6ql+2ysPRrtlbt2V8Zf697VsNX5DTgZS9BKmWlAeqejNYaqG5Rsuhn7szbCfkkmsjedk+Rdcv A3SHMBeHXdtfBHS0AlbEwKgeml08NmCUcwnifhrQywCnu8NN9+RQ3cUxGvIuLLSzi3915wC6hbxr 8xArckQfSUfKA/hrHvoiiCGZU9D23xj3XXtsjdbIIDXATDnCPrKANdvGN5LTKal8bT0jXORfAz1z MniqVUgvWVNcviPgQ9BfT5qpGo8g7LaoBMGamAGVX6Ezrs04rk8jQ1yz1bB/8URfTRLZdyYkMh0u MB4xMylnyavgusoi7Duf5RuYJvNaL0g8Lx/cfGpGsGwdD2Lj/qRC45ammn6wCxDVfiJV6Z/TzJcY PBvzWK5xT++PQgMV8EwtXwA1kFqaGrcuiDHejMQ8O82Edjr+eBCBe0B7bRddoMD6oOlhNm1YsSNt [vdsm@node02:] $ tail file.txt 9JX8OWCJwbyvEPDyyI30H1/jPZfDo1sS11dZ2JjiO7qhB45VaU8+irG45D0GGJhFf8wE8TD9EGWG 8346QHLX9ZSFsbjpuh71hr5Ju1UduVdvIDwwP8WDBtRUbMAVvsyGR33rkpijepmUjmYl/jeZ7rsC VyUVlmG5PxrI7KKxz5dSkzApqVHKKgsf93JMDAdPwvXTq4hhZdUJ581w9FC/f9k2wWldEGkAcyB0 cCKp+VJl2vx989KUoqAJzsrvYdK0X7itruqYdpC29JXode+7NixUflhKvPdKmitBYyCEgCcyxUyn eyMOdaan2x8d8MztLLoWLpp+gLzl2Hev7y3OXq6I9SVN2t+hcVIz8Llmumy0cD+VC4u2/UZszYqS nDaSSMs35agGUUgIpHjPxCRf/yqnfrJJMTGAcxSEqHtpEdsjEmkf4QkyEgEZ13f4oi7P/DFCIIvV JBsHzOLDoetnFzAA2/RqbDflPrVWcAR7tXVqGLACCj2s19uUFSNb8nBWmEk8fFz31iJhuL43v0WE 78/THl49T0hhzHQp6kdIiw5p1zPUIFGBZ0BS4mBCHxu+tMlPZe1zWJMJZdPnvDNtHZ4gQ6LFgU4w E16ACZaLqLhx2oUUUov5JHvQcVFohn6HH+eog6XZCiTaG0Tue 4 Feb 13:18:37 CET 2020 [vdsm@node02:] $ dd if=file.txt of=/dev/null dd: error reading ‘file.txt’: Permission denied 131072+0 records in 131072+0 records out 67108864 bytes (67 MB) copied, 0.106097 s, 633 MB/s if root does dd first, all is peachy: [root@node02] # dd if=file.txt of=/dev/null 390625+1 records in 390625+1 records out 200000058 bytes (200 MB) copied, 0.345906 s, 578 MB/s [vdsm@node02] $ dd if=file.txt of=/dev/null 390625+1 records in 390625+1 records out 200000058 bytes (200 MB) copied, 0.188451 s, 1.1 GB/s Error in the gluster.log: [2020-02-04 12:27:57.915356] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ssd_storage-client-1: remote operation failed. Path: /.shard/57200f4f-537d-4e56-9258-38fe6ac64c4e.2 (00000000-0000-0000-0000-000000000000) [Permission denied] [2020-02-04 12:27:57.915404] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ssd_storage-client-0: remote operation failed. Path: /.shard/57200f4f-537d-4e56-9258-38fe6ac64c4e.2 (00000000-0000-0000-0000-000000000000) [Permission denied] [2020-02-04 12:27:57.915472] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ssd_storage-client-2: remote operation failed. Path: /.shard/57200f4f-537d-4e56-9258-38fe6ac64c4e.2 (00000000-0000-0000-0000-000000000000) [Permission denied] [2020-02-04 12:27:57.915490] E [MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-ssd_storage-shard: Lookup on shard 2 failed. Base file gfid = 57200f4f-537d-4e56-9258-38fe6ac64c4e [Permission denied] What we tried: - restarting single hosts, - restarting the entire cluster, - doing stuff like find /rhev .. exec stats{}\ ; - dd'ing (read) all of the mount dir... We are out of ideas and also no experts on either gluster nor ovirt, it seems. And this is supposed to be a production HA environment. Any help would be appreciated. I hope I did think of all the relevant data and logs. -- with kind regards, mit freundlichen Gruessen, Christian Reiss

3 13

oVirt behavior with thin provision/deduplicated block storage
by Alan G 24 Feb '20

24 Feb '20

Hi, I have an oVirt cluster with a storage domain hosted on a FC storage array that utilises block de-duplication technology. oVirt reports the capacity of the domain as though the de-duplication factor was 1:1, which of course is not the case. So what I would like to understand is the likely behavior of oVirt when the used space approaches the reported capacity. Particularly around the critical action space blocker. Thanks, Alan

4 7

New Host Migration Failure & Console Failure
by Jonathan Mathews 24 Feb '20

24 Feb '20

Good Day I have installed new oVirt platform with hosted engine, but when I add a new host, the notification keeps repeating "Finished Activating Host" and it does not stop until I select do not disturb for 1 day. (Screenshot attached) Also once the new host has been added, I am unable to migrate a VM to it, if I start a VM up on the new host, I am unable to migrate the VM away from it or launch a console. Please see the following logs from when I tried to migrate a VM to the new host. 2020-02-19 09:48:27,034Z INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-55) [0de8c750-2955-40f3-ab75-a33d29894199] Running command: MigrateVmToServerCommand internal: false. Entities affected : ID: 7b0b6e6d-d099-43e0-933f-3c335b54a3a1 Type: VMAction group MIGRATE_VM with role type USER 2020-02-19 09:48:27,113Z INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (default task-55) [0de8c750-2955-40f3-ab75-a33d29894199] START, MigrateVDSCommand( MigrateVDSCommandParameters:{hostId='df715653-daf4-457e-839d-95683ab21234', vmId='7b0b6e6d-d099-43e0-933f-3c335b54a3a1', srcHost=' host03.timefreight.co.za', dstVdsId='896b7f02-00e9-405c-b166-ec103a7f9ee8', dstHost='host01.timefreight.co.za:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='62', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]', dstQemu='172.10.10.1'}), log id: 44b23456 2020-02-19 09:48:27,114Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (default task-55) [0de8c750-2955-40f3-ab75-a33d29894199] START, MigrateBrokerVDSCommand(HostName = host03.timefreight.co.za, MigrateVDSCommandParameters:{hostId='df715653-daf4-457e-839d-95683ab21234', vmId='7b0b6e6d-d099-43e0-933f-3c335b54a3a1', srcHost=' host03.timefreight.co.za', dstVdsId='896b7f02-00e9-405c-b166-ec103a7f9ee8', dstHost='host01.timefreight.co.za:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='62', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]', dstQemu='172.10.10.1'}), log id: 566212f6 2020-02-19 09:48:27,122Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (default task-55) [0de8c750-2955-40f3-ab75-a33d29894199] FINISH, MigrateBrokerVDSCommand, return: , log id: 566212f6 2020-02-19 09:48:27,126Z INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (default task-55) [0de8c750-2955-40f3-ab75-a33d29894199] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 44b23456 2020-02-19 09:48:27,138Z INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-10) [] VM '7b0b6e6d-d099-43e0-933f-3c335b54a3a1'(accpac) moved from 'MigratingFrom' --> 'Up' 2020-02-19 09:48:27,138Z INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-10) [] Adding VM '7b0b6e6d-d099-43e0-933f-3c335b54a3a1'(accpac) to re-run list 2020-02-19 09:48:27,141Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-10) [] Rerun VM '7b0b6e6d-d099-43e0-933f-3c335b54a3a1'. Called from VDS ' host03.timefreight.co.za' 2020-02-19 09:48:27,143Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-55) [0de8c750-2955-40f3-ab75-a33d29894199] EVENT_ID: VM_MIGRATION_START(62), Migration started (VM: accpac, Source: host03.timefreight.co.za, Destination: host01.timefreight.co.za, User: admin@internal-authz). 2020-02-19 09:48:27,199Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-92544) [] START, MigrateStatusVDSCommand(HostName = host03.timefreight.co.za, MigrateStatusVDSCommandParameters:{hostId='df715653-daf4-457e-839d-95683ab21234', vmId='7b0b6e6d-d099-43e0-933f-3c335b54a3a1'}), log id: 4978bfea 2020-02-19 09:48:27,203Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-92544) [] FINISH, MigrateStatusVDSCommand, return: , log id: 4978bfea 2020-02-19 09:48:27,242Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-92544) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: accpac, Source: host03.timefreight.co.za, Destination: host01.timefreight.co.za). 2020-02-19 09:48:27,254Z INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (EE-ManagedThreadFactory-engine-Thread-92544) [] Lock freed to object 'EngineLock:{exclusiveLocks='[7b0b6e6d-d099-43e0-933f-3c335b54a3a1=VM]', sharedLocks=''}' The first host is running oVirt 4.3.7, the Hosted-Engine is running oVirt 4.3.8 and the new host is running oVirt 4.3.8 I would really appreciate some assistance. Sincerely Jonathan Mathews

1 1

Re: iSCSI Domain Addition Fails
by eevans＠digitaldatatechs.com 23 Feb '20

23 Feb '20

Do you have a lun associated with the target? Eric Evans Digital Data Services LLC. 304.660.9080 From: Robert Webb <rwebb(a)ropeguru.com> Sent: Sunday, February 23, 2020 9:06 AM To: users(a)ovirt.org Subject: [ovirt-users] iSCSI Domain Addition Fails So I am messing around with FreeNAS and iSCSI. FreeNAS has a target configured, it is discoverable in oVirt, but then I click "OK" nothing happens. I have a name for the domain defined and have expanded the advanced features, but cannot find it anything showing an error. oVirt 4.3.8

2 1

Re: iSCSI Domain Addition Fails
by Benny Zlotnik 23 Feb '20

23 Feb '20

anything in the vdsm or engine logs? On Sun, Feb 23, 2020 at 4:23 PM Robert Webb <rwebb(a)ropeguru.com> wrote: > > Also, I did do the “Login” to connect to the target without issue, from what I can tell. > > > > From: Robert Webb > Sent: Sunday, February 23, 2020 9:06 AM > To: users(a)ovirt.org > Subject: iSCSI Domain Addition Fails > > > > So I am messing around with FreeNAS and iSCSI. FreeNAS has a target configured, it is discoverable in oVirt, but then I click “OK” nothing happens. > > > > I have a name for the domain defined and have expanded the advanced features, but cannot find it anything showing an error. > > > > oVirt 4.3.8 > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FMAXFDMNHVGTM…

1 0

Re: iSCSI Domain Addition Fails
by Robert Webb 23 Feb '20

23 Feb '20

Also, I did do the "Login" to connect to the target without issue, from what I can tell. From: Robert Webb Sent: Sunday, February 23, 2020 9:06 AM To: users(a)ovirt.org Subject: iSCSI Domain Addition Fails So I am messing around with FreeNAS and iSCSI. FreeNAS has a target configured, it is discoverable in oVirt, but then I click "OK" nothing happens. I have a name for the domain defined and have expanded the advanced features, but cannot find it anything showing an error. oVirt 4.3.8

1 0

iSCSI Domain Addition Fails
by Robert Webb 23 Feb '20

23 Feb '20

So I am messing around with FreeNAS and iSCSI. FreeNAS has a target configured, it is discoverable in oVirt, but then I click "OK" nothing happens. I have a name for the domain defined and have expanded the advanced features, but cannot find it anything showing an error. oVirt 4.3.8

1 0

dashboard data empty
by Fabrizio 23 Feb '20

23 Feb '20

Hello everyone. I intalled 4 node with hosted engine, everything works fine exept the dashboard. it doesn't collect any data and is always empty of information. searching around I found out that the problem could be the dwh service, inside the log there is an error: ## log file 2020-02-19 17:46:14|ETL Service Started ovirtEngineDbDriverClass|org.postgresql.Driver ovirtEngineHistoryDbJdbcConnection|jdbc:postgresql://localhost:5432/ovirt_engine_history?sslfactory=org.postgresql.ssl.NonValidatingFactory hoursToKeepDaily|0 hoursToKeepHourly|720 ovirtEngineDbPassword|********************** runDeleteTime|3 ovirtEngineDbJdbcConnection|jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory runInterleave|60 limitRows|limit 1000 ovirtEngineHistoryDbUser|ovirt_engine_history ovirtEngineDbUser|engine deleteIncrement|10 timeBetweenErrorEvents|300000 hoursToKeepSamples|24 deleteMultiplier|1000 lastErrorSent|2011-07-03 12:46:47.000000 etlVersion|4.3.8 dwhAggregationDebug|false dwhUuid|5c4ffc1b-928f-4d06-a40f-bb34ba11c7f2 ovirtEngineHistoryDbDriverClass|org.postgresql.Driver ovirtEngineHistoryDbPassword|********************** Exception in component tJDBCInput_2 org.postgresql.util.PSQLException: Bad value for type short : 201905 at org.postgresql.jdbc2.AbstractJdbc2ResultSet.getShort(AbstractJdbc2ResultSet.java:2098) at ovirt_engine_dwh.osenumupdate_4_3.OsEnumUpdate.tJDBCInput_2Process(OsEnumUpdate.java:2354) at ovirt_engine_dwh.osenumupdate_4_3.OsEnumUpdate.tRowGenerator_1Process(OsEnumUpdate.java:1921) at ovirt_engine_dwh.osenumupdate_4_3.OsEnumUpdate.tJDBCInput_4Process(OsEnumUpdate.java:1376) at ovirt_engine_dwh.osenumupdate_4_3.OsEnumUpdate.tJDBCConnection_1Process(OsEnumUpdate.java:873) at ovirt_engine_dwh.osenumupdate_4_3.OsEnumUpdate.tJDBCConnection_2Process(OsEnumUpdate.java:736) at ovirt_engine_dwh.osenumupdate_4_3.OsEnumUpdate.runJobInTOS(OsEnumUpdate.java:4396) at ovirt_engine_dwh.osenumupdate_4_3.OsEnumUpdate.runJob(OsEnumUpdate.java:4099) at ovirt_engine_dwh.samplerunjobs_4_3.SampleRunJobs.tRunJob_4Process(SampleRunJobs.java:947) at ovirt_engine_dwh.samplerunjobs_4_3.SampleRunJobs.tJDBCConnection_2Process(SampleRunJobs.java:767) at ovirt_engine_dwh.samplerunjobs_4_3.SampleRunJobs.tJDBCConnection_1Process(SampleRunJobs.java:642) at ovirt_engine_dwh.samplerunjobs_4_3.SampleRunJobs$2.run(SampleRunJobs.java:2683) ## SERVICE STATUS ● ovirt-engine-dwhd.service - oVirt Engine Data Warehouse Loaded: loaded (/usr/lib/systemd/system/ovirt-engine-dwhd.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2020-02-19 17:46:14 CET; 16h ago Main PID: 6425 (ovirt-engine-dw) CGroup: /system.slice/ovirt-engine-dwhd.service ├─6425 /usr/bin/python /usr/share/ovirt-engine-dwh/services/ovirt-engine-dwhd/ovirt-engine-dwhd.py --redirect-output --systemd=notify start └─6512 ovirt-engine-dwhd -Dorg.ovirt.engine.dwh.settings=/tmp/tmparNn_x/settings.properties -Xms1g -Xmx1g -classpath /usr/share/ovirt-engine-dwh/lib/*::/usr/share/java/dom4j.jar:/usr/share/java/apache-commons-collections.jar:/usr/share/java/postgresql-jdbc.jar ovirt_engine_dwh.historyetl_4_3.HistoryETL --context=Default Feb 19 17:46:14 engine.ovirt.local systemd[1]: Starting oVirt Engine Data Warehouse... Feb 19 17:46:14 engine.ovirt.local systemd[1]: Started oVirt Engine Data Warehouse.

4 4

metrics-store installation
by Christoph Köhler 23 Feb '20

23 Feb '20

Hello! I have installed metrics store along this documentation: https://ovirt.org/documentation/metrics-install-guide/Installing_Metrics_St… Everything went fine without errors, so I have now the login page in my browser. The Problem is that the credentials for login seems to be not taken over. The user should be 'admin' and the password is given at the beginning in secure_vars.yaml. Does anyone have an idea what to do now? Greetings! Chris

2 1

paused vm's will not resume
by eevans＠digitaldatatechs.com 23 Feb '20

23 Feb '20

I have 2 vm's, which are the most important in my world, that paused and will not resume. I have googled this to death but no solution. It stated a lack of space but none of the drives on my hosts are using more than 30% or there space and these 2 have ran on kvm host for several years and always had at least 50% free space. I like ovirt and want to use it but I cannot tolerate the down time. If I cannot get this resolved, I'm going back to kvm hosts. I am pulling my hair out here. If anyone can help with this issue, please let me know.

6 10

Hugepages and running out of memory
by klaasdemter＠gmail.com 22 Feb '20

22 Feb '20

Hi, this e-mail is meant to caution people using hugepages and mixing those VMs with non-hugepages VMs on the same hypervisors. I ran into major trouble, the hypervisors ran out of memory because the VM scheduling disregards the hugepages in it's calculations. So if you have hugepages and non-hugepages VMs better check the memory commited on a hypervisor manually :) https://bugzilla.redhat.com/show_bug.cgi?id=1804037 https://bugzilla.redhat.com/show_bug.cgi?id=1804046 As for workarounds: so far it seems the only viable solution is splitting hugepages/nonhugepages VMs with affinity groups but at least for me that means wasting a lot of resources. Greetings Klaas

1 0

main channel: failed to connect HTTP proxy connection not allowed
by Jorick Astrego 21 Feb '20

21 Feb '20

Hi, Having a spice console issue on our new 4.3.8 cluster. The console stays blank and with the cli and debug on, I get the following error "main channel: failed to connect HTTP proxy connection not allowed" We have a userportal through haproxy so this is configured in ovirt engine engine-config -g SpiceProxyDefault SpiceProxyDefault: http://userportal.*.*:****/ version: general But the same setting works fine on our 4.2 cluster. I can telnet to the ports on the 4.3.8 hosts without issue. Non working console on newer cluster: remote-viewer -v --debug console.vv (remote-viewer:29267): virt-viewer-DEBUG: 16:24:24.697: Opening display to console.vv (remote-viewer:29267): virt-viewer-DEBUG: 16:24:24.700: Guest (null) has a spice display Guest (null) has a spice display (remote-viewer:29267): virt-viewer-DEBUG: 16:24:24.789: Spice foreign menu updated (remote-viewer:29267): virt-viewer-DEBUG: 16:24:24.790: After open connection callback fd=-1 (remote-viewer:29267): virt-viewer-DEBUG: 16:24:24.790: Opening connection to display at console.vv Opening connection to display at console.vv (remote-viewer:29267): virt-viewer-DEBUG: 16:24:24.791: fullscreen display 0: 0 (remote-viewer:29267): virt-viewer-DEBUG: 16:24:24.791: app is not in full screen (remote-viewer:29267): virt-viewer-DEBUG: 16:24:24.792: New spice channel 0x55704bf70d40 SpiceMainChannel 0 (remote-viewer:29267): virt-viewer-DEBUG: 16:24:24.793: notebook show status 0x55704b820280 (remote-viewer:29267): virt-viewer-DEBUG: 16:24:24.969: main channel: failed to connect HTTP proxy connection not allowed (remote-viewer:29267): virt-viewer-DEBUG: 16:24:24.969: Destroy SPICE channel SpiceMainChannel 0 (remote-viewer:29267): virt-viewer-DEBUG: 16:24:24.969: zap main channel (remote-viewer:29267): virt-viewer-WARNING **: 16:24:24.969: Channel error: HTTP proxy connection not allowed Working console on older cluster: remote-viewer -v --debug console.vv (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.111: Opening display to console.vv (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.111: Guest (null) has a spice display Guest (null) has a spice display (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.209: Spice foreign menu updated (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.209: After open connection callback fd=-1 (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.209: Opening connection to display at console.vv Opening connection to display at console.vv (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.211: fullscreen display 0: 0 (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.211: app is not in full screen (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.212: New spice channel 0x556a03f7e820 SpiceMainChannel 0 (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.212: notebook show status 0x556a03c7a280 (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.397: main channel: opened (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.397: notebook show status 0x556a03c7a280 (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.404: virt_viewer_app_set_uuid_string: UUID changed to a2d64ffe-7583-45b6-92f3-87661039388c (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.404: app is not in full screen (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.502: app is not in full screen (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.514: New spice channel 0x556a043bb660 SpiceDisplayChannel 0 (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.514: New spice channel 0x556a0422b570 SpiceCursorChannel 0 (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.515: New spice channel 0x556a0422c000 SpiceInputsChannel 0 (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.515: new inputs channel (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.661: creating spice display (#:0) (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.661: Insert display 0 0x556a03ca29e0 (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.699: creating spice display (#:1) (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.699: Insert display 1 0x556a03ca2830 (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.699: creating spice display (#:2) (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.700: Insert display 2 0x556a03ca2680 (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.700: creating spice display (#:3) (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.700: Insert display 3 0x556a03ca24d0 (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.701: Found a window without a display, reusing for display #0 (remote-viewer:29334): virt-viewer-DEBUG: 16:25:59.701: notebook show display 0x556a03c7a280 (remote-viewer:29334): virt-viewer-DEBUG: 16:26:00.074: Allocated 1024x768 (remote-viewer:29334): virt-viewer-DEBUG: 16:26:00.074: Child allocate 1024x768 (remote-viewer:29334): virt-viewer-DEBUG: 16:26:04.846: Allocated 1024x768 (remote-viewer:29334): virt-viewer-DEBUG: 16:26:04.846: Child allocate 1024x768 Met vriendelijke groet, With kind regards, Jorick Astrego Netbulae Virtualization Experts ---------------- Tel: 053 20 30 270 info(a)netbulae.eu Staalsteden 4-3A KvK 08198180 Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01 ----------------

1 1

[ANN] oVirt 4.3.9 Second Release Candidate is now available for testing
by Lev Veyde 21 Feb '20

21 Feb '20

The oVirt Project is pleased to announce the availability of the oVirt 4.3.9 Second Release Candidate for testing, as of February 20th, 2020. This update is a release candidate of the ninth in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used in production. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) * oVirt Node 4.3 (available for x86_64 only) has been built consuming CentOS 7.7 Release See the release notes [1] for known issues, new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Node is already available Additional Resources: * Read more about the oVirt 4.3.9 release highlights: http://www.ovirt.org/release/4.3.9/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.9/ [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/ -- Lev Veyde Senior Software Engineer, RHCE | RHCVA | MCITP Red Hat Israel <https://www.redhat.com> lev(a)redhat.com | lveyde(a)redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

3 2

Understanding dashboard memory data
by nux＠li.nux.ro 21 Feb '20

21 Feb '20

Hello, I'm having to deal with an oVirt installation, once I log in the dashboard says: Memory: "1.2 Available of 4.5 TiB" and "Virtual resources - Committed: 113%, Allocated: 114%". So, clearly there is RAM available, but what's with the committed and allocated numbers? Regards

1 0

Ovirt storage domain ovirt-image-repository is unattached and can't use change CD to install windows on VM
by Heinrich Momberg 20 Feb '20

20 Feb '20

Good day, Whenever i am trying to use the change CD to mount the ovirt tools iso so i can install windows server or any windows it says "could not find ISO file collection".It doesn't pick up the hdd that i have created in the windwos setup so i have to do the whole ovirt tools mount and select the windows i want to insteall etc. It has been working fine until a couple of days ago when we had power outages. I also noticed that the storage domain,ovirt-image-repository is unattached... How do i go about fixing this problem or where can i check logs. I have checked the permissions where the ISO's or images are stored.

3 9

Spacewalk Integration
by eevans＠digitaldatatechs.com 20 Feb '20

20 Feb '20

I finally figured out how to use Spacewalk as an external provider in Ovirt. For the URL I used http://sat.digitaldatatechs.com/XMLRPC and used the admin login credentials. It registered and Spacewalk has the capability of provisioning a bare bones machine. I have not tested provisioning but I will comment once I do.

1 0

Get CPU Family
by Luca 20 Feb '20

20 Feb '20

Hello guys, how can I get the correct CPU Family in a ovirt host? When I execute virsh -r capabilities I get only "Broadwell-noTSX-IBRS" and not the full "Broadwell-noTSX IBRS SSBD MDS" Regards, Luca

3 2

Re: Add LDAP user : ERROR: null value in column "external_id" violates not-null constraint
by Lucas Lamy 20 Feb '20

20 Feb '20

Anybody ?

2 1

Unsynced entries do not self-heal during upgrade from oVirt 4.2 -> 4.3
by Goorkate, B.J. 20 Feb '20

20 Feb '20

Hi all, I'm in the process of upgrading oVirt-nodes from 4.2 to 4.3. After upgrading the first of 3 oVirt/gluster nodes, there are between 600-1200 unsynced entries for a week now on 1 upgraded node and one not-yet-upgraded node. The third node (also not-yet-upgraded) says it's OK (no unsynced entries). The cluster doesn't seem to be very busy, but somehow self-heal doesn't complete. Is this because of different gluster versions across the nodes and will it resolve as soon as I upgraded all nodes? Since it's our production cluster, I don't want to take any risk... Does anybody recognise this problem? Of course I can provide more information if necessary. Any hints on troubleshooting the unsynced entries are more than welcome! Thanks in advance! Regards, Bertjan ------------------------------------------------------------------------------ De informatie opgenomen in dit bericht kan vertrouwelijk zijn en is uitsluitend bestemd voor de geadresseerde. Indien u dit bericht onterecht ontvangt, wordt u verzocht de inhoud niet te gebruiken en de afzender direct te informeren door het bericht te retourneren. Het Universitair Medisch Centrum Utrecht is een publiekrechtelijke rechtspersoon in de zin van de W.H.W. (Wet Hoger Onderwijs en Wetenschappelijk Onderzoek) en staat geregistreerd bij de Kamer van Koophandel voor Midden-Nederland onder nr. 30244197. Denk s.v.p aan het milieu voor u deze e-mail afdrukt. ------------------------------------------------------------------------------ This message may contain confidential information and is intended exclusively for the addressee. If you receive this message unintentionally, please do not use the contents but notify the sender immediately by return e-mail. University Medical Center Utrecht is a legal person by public law and is registered at the Chamber of Commerce for Midden-Nederland under no. 30244197. Please consider the environment before printing this e-mail.

3 12

Re: Engine installation failed at - get local vm ip
by shawly 20 Feb '20

20 Feb '20

Still have this problem, log doesn't say anything but that it failed after 90 tries, no reason in the log. Using VMWare Workstation 14, guest is running oVirt Node 4.3.8.

2 1

Can't install virtio-win with EL7.7/Ovirt-4.3.8 -- rpm error
by Derek Atkins 19 Feb '20

19 Feb '20

Hi, I was trying to install the virtio-win package, but it gives an error: ERROR You need to update rpm to handle: rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by virtio-win-0.1.173-6.noarch Is this a known problem with current 4.3.x and EL7.7? -derek -- Derek Atkins 617-623-3745 derek(a)ihtfp.com www.ihtfp.com Computer and Internet Security Consultant

6 7

Re: populating the ISO domain
by Gianluca Cecchi 19 Feb '20

19 Feb '20

On Wed, Feb 19, 2020 at 4:37 PM Robert Webb <rwebb(a)ropeguru.com> wrote: > No. > > I am using the Spice viewer and my Data domain is on NFS. When I open the > "Change CD" option in the viewer, I only see ISO's from the ISO Domain. > > I haven't found any other way to actually mount an ISO to a VM other then > the viewer which kinda sucks. I am used to other environments where I could > just mount an ISO to a cdrom in the VM config and it shows up in the OS. > > If you can shutdown the VM, you could try to shutdown it and then choose the arrow near "run" and select "run once". There you can attach an iso: do you see your data domain isos there? In my case I have isos on block based data domains: I cannot dynamically change cd due to the bug I reported, but if I start a VM with "run once" I can see the ISOs on data domain available and usable (eg to install OS) See also the "Explanation of Settings in the Run Once Window" section here: https://www.ovirt.org/documentation/vmm-guide/appe-Reference_Settings_in_Ad… Gianluca

2 1

Re: populating the ISO domain
by Vinícius Ferrão 19 Feb '20

19 Feb '20

Hi. ISO domains were deprecated. Now you should use a Data Domain instead and fill with the ISOs. If you still using ISO Domain, like me, I only scp the ISOs directly to the storage. You don’t need to use the ISO uploader script. I never made it work anyway LOL. Sent from my iPhone > On 19 Feb 2020, at 04:33, "eevans(a)digitaldatatechs.com" <eevans(a)digitaldatatechs.com> wrote: >

3 2

Re: populating the ISO domain
by Gianluca Cecchi 19 Feb '20

19 Feb '20

On Wed, Feb 19, 2020 at 4:11 PM Robert Webb <rwebb(a)ropeguru.com> wrote: > So I had also read that ISO Domains have been deprecated but ran into an > issue when using console access. It seems that the client does not see any > ISO files from the Data domains when using the "Change CD" option and that > is the only option that I have found to be able to use when I need to mount > an ISO file to a VM. > > Am I missing something? If not, how can we deprecate the ISO Domain? > > In case of data domains on block based storage (iSCSI and FC) there is this outstanding bugzilla not fixed yet: https://bugzilla.redhat.com/show_bug.cgi?id=1589763 I don't know if it is your use case. Gianluca

3 2

Re: populating the ISO domain
by Strahil Nikolov 19 Feb '20

19 Feb '20

On February 19, 2020 5:37:28 PM GMT+02:00, Robert Webb <rwebb(a)ropeguru.com> wrote: >No. > >I am using the Spice viewer and my Data domain is on NFS. When I open >the "Change CD" option in the viewer, I only see ISO's from the ISO >Domain. > >I haven't found any other way to actually mount an ISO to a VM other >then the viewer which kinda sucks. I am used to other environments >where I could just mount an ISO to a cdrom in the VM config and it >shows up in the OS. > >________________________________________ >From: Gianluca Cecchi <gianluca.cecchi(a)gmail.com> >Sent: Wednesday, February 19, 2020 10:34 AM >To: Robert Webb >Cc: Vinícius Ferrão; eevans(a)digitaldatatechs.com; users(a)ovirt.org >Subject: Re: [ovirt-users] Re: populating the ISO domain > >On Wed, Feb 19, 2020 at 4:11 PM Robert Webb ><rwebb(a)ropeguru.com<mailto:rwebb@ropeguru.com>> wrote: >So I had also read that ISO Domains have been deprecated but ran into >an issue when using console access. It seems that the client does not >see any ISO files from the Data domains when using the "Change CD" >option and that is the only option that I have found to be able to use >when I need to mount an ISO file to a VM. > >Am I missing something? If not, how can we deprecate the ISO Domain? > > >In case of data domains on block based storage (iSCSI and FC) there is >this outstanding bugzilla not fixed yet: >https://bugzilla.redhat.com/show_bug.cgi?id=1589763 > >I don't know if it is your use case. > >Gianluca >_______________________________________________ >Users mailing list -- users(a)ovirt.org >To unsubscribe send an email to users-leave(a)ovirt.org >Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >oVirt Code of Conduct: >https://www.ovirt.org/community/about/community-guidelines/ >List Archives: >https://lists.ovirt.org/archives/list/users@ovirt.org/message/FYMZYKUUGQZHQ… What about going in Admin portal, select the VM, and then click the 3 dots button ? There should be a 'Eject' and all your ISOs (I have already migrated to data domain). Best Regards, Strahil Nikolov

2 1