Problems with RHEL 9.4 hosts
Are there any known incompatibilities with RHEL 9.4 (and derivatives)? We are running a 7-node ovirt 4.5.5-1.el8 self hosted engine cluster, with all of the hosts running AlmaLinux 9. After upgrading from 9.3 to 9.4, every node started flapping between “Up” and “NonOperational,” with VMs in turn migrating between hosts. I believe the underlying issue (or at least the point I got stuck at) was with two of our logical networks being stuck “out of sync” on all hosts. I was unable to synchronize networks or setup the networks using the UI. A reinstall of a host succeeded but then the host immediately reverted to the same state with the same networks being out of sync. I eventually found that if I downgraded the host from 9.4 to 9.3, it immediately became stable and back online. Are there any known incompatibilities with RHEL 9.4 (and derivatives)? If not, I’m happy to upgrade a single node to test. Please just let me know what log files and details would be most helpful in debugging what goes wrong. (And yes, I know we need to upgrade the hosted engine VM itself now that CentOS Stream 8 is now EOL). Many thanks, Devin
You most likely need the following patch: https://github.com/oVirt/vdsm/commit/49eaf70c5a14eb00e85eac5f91ac36f010a9a32... Test with that, guess it's fixed then :) On 4/06/2024 22:33, Devin A. Bougie wrote:
Are there any known incompatibilities with RHEL 9.4 (and derivatives)?
We are running a 7-node ovirt 4.5.5-1.el8 self hosted engine cluster, with all of the hosts running AlmaLinux 9. After upgrading from 9.3 to 9.4, every node started flapping between “Up” and “NonOperational,” with VMs in turn migrating between hosts.
I believe the underlying issue (or at least the point I got stuck at) was with two of our logical networks being stuck “out of sync” on all hosts. I was unable to synchronize networks or setup the networks using the UI. A reinstall of a host succeeded but then the host immediately reverted to the same state with the same networks being out of sync.
I eventually found that if I downgraded the host from 9.4 to 9.3, it immediately became stable and back online.
Are there any known incompatibilities with RHEL 9.4 (and derivatives)? If not, I’m happy to upgrade a single node to test. Please just let me know what log files and details would be most helpful in debugging what goes wrong.
(And yes, I know we need to upgrade the hosted engine VM itself now that CentOS Stream 8 is now EOL).
Many thanks, Devin
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PIROQCVLPVBNJY...
Thanks so much! Yes, that patch fixed the “out of sync network” issue. However, we’re still unable to join a fully updated 9.4 host to the cluster - now with "Failed to connect Host to Storage Servers”. Downgrading all of the updated packages fixes the issue. Please see the attached vdsm.log and supervdsm.log from the host after updating it to EL 9.4 and then trying to activate it. Any more suggestions would be greatly appreciated. Thanks again, Devin
On Jun 5, 2024, at 2:35 AM, Jean-Louis Dupond <jean-louis@dupond.be> wrote:
Test with that, guess it's fixed then :)
On 4/06/2024 22:33, Devin A. Bougie wrote:
Are there any known incompatibilities with RHEL 9.4 (and derivatives)?
We are running a 7-node ovirt 4.5.5-1.el8 self hosted engine cluster, with all of the hosts running AlmaLinux 9. After upgrading from 9.3 to 9.4, every node started flapping between “Up” and “NonOperational,” with VMs in turn migrating between hosts.
I believe the underlying issue (or at least the point I got stuck at) was with two of our logical networks being stuck “out of sync” on all hosts. I was unable to synchronize networks or setup the networks using the UI. A reinstall of a host succeeded but then the host immediately reverted to the same state with the same networks being out of sync.
I eventually found that if I downgraded the host from 9.4 to 9.3, it immediately became stable and back online.
Are there any known incompatibilities with RHEL 9.4 (and derivatives)? If not, I’m happy to upgrade a single node to test. Please just let me know what log files and details would be most helpful in debugging what goes wrong.
(And yes, I know we need to upgrade the hosted engine VM itself now that CentOS Stream 8 is now EOL).
Many thanks, Devin
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fprivacy-policy.html&data=05%7C02%7Cdevin.bougie%40cornell.edu%7Cebf2c01b70a742e81ba008dc8529b2a0%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638531661341449418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=2CeQ8D3nliRAocn6M1%2Fe%2FYYrkswTgUwt2us5qLpaLVQ%3D&reserved=0<https://www.ovirt.org/privacy-policy.html> oVirt Code of Conduct: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fcommunity%2Fabout%2Fcommunity-guidelines%2F&data=05%7C02%7Cdevin.bougie%40cornell.edu%7Cebf2c01b70a742e81ba008dc8529b2a0%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638531661341456829%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=%2F3%2BQ4XtHx5ntfJeamOcxEhn5YseHcluJLuSN9N4%2FfZc%3D&reserved=0<https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovirt.org%2Farchives%2Flist%2Fusers%40ovirt.org%2Fmessage%2FPIROQCVLPVBNJY6FWLE3VSLHRAZKRB3L%2F&data=05%7C02%7Cdevin.bougie%40cornell.edu%7Cebf2c01b70a742e81ba008dc8529b2a0%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638531661341461702%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=S7WhPZ%2BEBVMC9mx6aInbMdOOcd1vXtWtJIyT5Ggv13o%3D&reserved=0<https://lists.ovirt.org/archives/list/users@ovirt.org/message/PIROQCVLPVBNJY6FWLE3VSLHRAZKRB3L/>
2024-06-06 13:28:10,478-0400 ERROR (jsonrpc/5) [storage.storageServer] Could not configure connection to 192.168.56.57:3260,1 iqn.2002-10.com.infortrend:raid.sn8087428.112 and iface <IscsiInterface name='bond1' transport='tcp' netIfaceName='bond1'>: (7, b'', b'iscsiadm: Error while adding record: invalid parameter\n') (storageServer:580) Seems like some issue with iscsiadm calls. Might want to debug which calls it does or what version change there is for iscsiadm. "Devin A. Bougie" <devin.bougie@cornell.edu> schreef op 6 juni 2024 19:32:29 CEST:
Thanks so much! Yes, that patch fixed the “out of sync network” issue. However, we’re still unable to join a fully updated 9.4 host to the cluster - now with "Failed to connect Host to Storage Servers”. Downgrading all of the updated packages fixes the issue.
Please see the attached vdsm.log and supervdsm.log from the host after updating it to EL 9.4 and then trying to activate it. Any more suggestions would be greatly appreciated.
Thanks again, Devin
On Jun 5, 2024, at 2:35 AM, Jean-Louis Dupond <jean-louis@dupond.be> wrote:
Test with that, guess it's fixed then :)
On 4/06/2024 22:33, Devin A. Bougie wrote:
Are there any known incompatibilities with RHEL 9.4 (and derivatives)?
We are running a 7-node ovirt 4.5.5-1.el8 self hosted engine cluster, with all of the hosts running AlmaLinux 9. After upgrading from 9.3 to 9.4, every node started flapping between “Up” and “NonOperational,” with VMs in turn migrating between hosts.
I believe the underlying issue (or at least the point I got stuck at) was with two of our logical networks being stuck “out of sync” on all hosts. I was unable to synchronize networks or setup the networks using the UI. A reinstall of a host succeeded but then the host immediately reverted to the same state with the same networks being out of sync.
I eventually found that if I downgraded the host from 9.4 to 9.3, it immediately became stable and back online.
Are there any known incompatibilities with RHEL 9.4 (and derivatives)? If not, I’m happy to upgrade a single node to test. Please just let me know what log files and details would be most helpful in debugging what goes wrong.
(And yes, I know we need to upgrade the hosted engine VM itself now that CentOS Stream 8 is now EOL).
Many thanks, Devin
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fprivacy-policy.html&data=05%7C02%7Cdevin.bougie%40cornell.edu%7Cebf2c01b70a742e81ba008dc8529b2a0%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638531661341449418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=2CeQ8D3nliRAocn6M1%2Fe%2FYYrkswTgUwt2us5qLpaLVQ%3D&reserved=0<https://www.ovirt.org/privacy-policy.html> oVirt Code of Conduct: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fcommunity%2Fabout%2Fcommunity-guidelines%2F&data=05%7C02%7Cdevin.bougie%40cornell.edu%7Cebf2c01b70a742e81ba008dc8529b2a0%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638531661341456829%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=%2F3%2BQ4XtHx5ntfJeamOcxEhn5YseHcluJLuSN9N4%2FfZc%3D&reserved=0<https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovirt.org%2Farchives%2Flist%2Fusers%40ovirt.org%2Fmessage%2FPIROQCVLPVBNJY6FWLE3VSLHRAZKRB3L%2F&data=05%7C02%7Cdevin.bougie%40cornell.edu%7Cebf2c01b70a742e81ba008dc8529b2a0%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638531661341461702%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=S7WhPZ%2BEBVMC9mx6aInbMdOOcd1vXtWtJIyT5Ggv13o%3D&reserved=0<https://lists.ovirt.org/archives/list/users@ovirt.org/message/PIROQCVLPVBNJY6FWLE3VSLHRAZKRB3L/>
Awesome, thanks again. Yes, the host is fixed by just downgrading the iscsi-initiator-utils and iscsi-initiator-utils-iscsiuio packages from: 6.2.1.9-1.gita65a472.el9.x86_64 to: 6.2.1.4-3.git2a8f9d8.el9.x86_64 Any additional pointers of where to look or how to debug the iscsiadm calls would be greatly appreciated. Many thanks! Devin
On Jun 6, 2024, at 2:04 PM, Jean-Louis Dupond <jean-louis@dupond.be> wrote:
2024-06-06 13:28:10,478-0400 ERROR (jsonrpc/5) [storage.storageServer] Could not configure connection to 192.168.56.57:3260,1 iqn.2002-10.com.infortrend:raid.sn8087428.112 and iface <IscsiInterface name='bond1' transport='tcp' netIfaceName='bond1'>: (7, b'', b'iscsiadm: Error while adding record: invalid parameter\n') (storageServer:580)
Seems like some issue with iscsiadm calls. Might want to debug which calls it does or what version change there is for iscsiadm.
"Devin A. Bougie" <devin.bougie@cornell.edu> schreef op 6 juni 2024 19:32:29 CEST: Thanks so much! Yes, that patch fixed the “out of sync network” issue. However, we’re still unable to join a fully updated 9.4 host to the cluster - now with "Failed to connect Host to Storage Servers”. Downgrading all of the updated packages fixes the issue.
Please see the attached vdsm.log and supervdsm.log from the host after updating it to EL 9.4 and then trying to activate it. Any more suggestions would be greatly appreciated.
Thanks again, Devin
On Jun 5, 2024, at 2:35 AM, Jean-Louis Dupond <jean-louis@dupond.be> wrote:
You most likely need the following patch: https://github.com/oVirt/vdsm/commit/49eaf70c5a14eb00e85eac5f91ac36f010a9a32...
Test with that, guess it's fixed then :)
On 4/06/2024 22:33, Devin A. Bougie wrote:
Are there any known incompatibilities with RHEL 9.4 (and derivatives)?
We are running a 7-node ovirt 4.5.5-1.el8 self hosted engine cluster, with all of the hosts running AlmaLinux 9. After upgrading from 9.3 to 9.4, every node started flapping between “Up” and “NonOperational,” with VMs in turn migrating between hosts.
I believe the underlying issue (or at least the point I got stuck at) was with two of our logical networks being stuck “out of sync” on all hosts. I was unable to synchronize networks or setup the networks using the UI. A reinstall of a host succeeded but then the host immediately reverted to the same state with the same networks being out of sync.
I eventually found that if I downgraded the host from 9.4 to 9.3, it immediately became stable and back online.
Are there any known incompatibilities with RHEL 9.4 (and derivatives)? If not, I’m happy to upgrade a single node to test. Please just let me know what log files and details would be most helpful in debugging what goes wrong.
(And yes, I know we need to upgrade the hosted engine VM itself now that CentOS Stream 8 is now EOL).
Many thanks, Devin
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PIROQCVLPVBNJY...
Weird, I have the same 6.2.1.9-1 version, and here it works. You can try to add some print here: https://github.com/oVirt/vdsm/blob/4d11cae0b1b7318b282d9f90788748c0ef3cc965/... This should print all executed iscsiadm commands. On 6/06/2024 20:50, Devin A. Bougie wrote:
Awesome, thanks again. Yes, the host is fixed by just downgrading the iscsi-initiator-utils and iscsi-initiator-utils-iscsiuio packages from: 6.2.1.9-1.gita65a472.el9.x86_64 to: 6.2.1.4-3.git2a8f9d8.el9.x86_64
Any additional pointers of where to look or how to debug the iscsiadm calls would be greatly appreciated.
Many thanks! Devin
On Jun 6, 2024, at 2:04 PM, Jean-Louis Dupond <jean-louis@dupond.be> wrote:
2024-06-06 13:28:10,478-0400 ERROR (jsonrpc/5) [storage.storageServer] Could not configure connection to 192.168.56.57:3260,1 iqn.2002-10.com.infortrend:raid.sn8087428.112 and iface <IscsiInterface name='bond1' transport='tcp' netIfaceName='bond1'>: (7, b'', b'iscsiadm: Error while adding record: invalid parameter\n') (storageServer:580)
Seems like some issue with iscsiadm calls. Might want to debug which calls it does or what version change there is for iscsiadm.
"Devin A. Bougie" <devin.bougie@cornell.edu> schreef op 6 juni 2024 19:32:29 CEST: Thanks so much! Yes, that patch fixed the “out of sync network” issue. However, we’re still unable to join a fully updated 9.4 host to the cluster - now with "Failed to connect Host to Storage Servers”. Downgrading all of the updated packages fixes the issue.
Please see the attached vdsm.log and supervdsm.log from the host after updating it to EL 9.4 and then trying to activate it. Any more suggestions would be greatly appreciated.
Thanks again, Devin
On Jun 5, 2024, at 2:35 AM, Jean-Louis Dupond <jean-louis@dupond.be> wrote:
You most likely need the following patch: https://github.com/oVirt/vdsm/commit/49eaf70c5a14eb00e85eac5f91ac36f010a9a32...
Test with that, guess it's fixed then :)
On 4/06/2024 22:33, Devin A. Bougie wrote:
Are there any known incompatibilities with RHEL 9.4 (and derivatives)?
We are running a 7-node ovirt 4.5.5-1.el8 self hosted engine cluster, with all of the hosts running AlmaLinux 9. After upgrading from 9.3 to 9.4, every node started flapping between “Up” and “NonOperational,” with VMs in turn migrating between hosts.
I believe the underlying issue (or at least the point I got stuck at) was with two of our logical networks being stuck “out of sync” on all hosts. I was unable to synchronize networks or setup the networks using the UI. A reinstall of a host succeeded but then the host immediately reverted to the same state with the same networks being out of sync.
I eventually found that if I downgraded the host from 9.4 to 9.3, it immediately became stable and back online.
Are there any known incompatibilities with RHEL 9.4 (and derivatives)? If not, I’m happy to upgrade a single node to test. Please just let me know what log files and details would be most helpful in debugging what goes wrong.
(And yes, I know we need to upgrade the hosted engine VM itself now that CentOS Stream 8 is now EOL).
Many thanks, Devin
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PIROQCVLPVBNJY...
Thank you! I added a warning at the line you indicated, which produces the following output: ——— /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,452-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,493-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,532-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,565-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,595-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,636-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,670-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,720-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,751-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2002-10.com.infortrend:raid.sn8087428.012', '-I', 'bond1', '-p', '192.168.56.55:3260,1', '--op=new'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,785-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,825-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2002-10.com.infortrend:raid.sn8073743.001', '-I', 'bond1', '-p', '192.168.56.54:3260,1', '--op=new'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,856-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,889-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2002-10.com.infortrend:raid.sn8073743.101', '-I', 'bond1', '-p', '192.168.56.56:3260,1', '--op=new'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,924-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,957-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2002-10.com.infortrend:raid.sn8087428.112', '-I', 'bond1', '-p', '192.168.56.57:3260,1', '--op=new'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:16,987-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:17,018-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2002-10.com.infortrend:raid.uid58204.012', '-I', 'bond1', '-p', '192.168.56.51:3260,1', '--op=new'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:17,051-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:17,079-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2002-10.com.infortrend:raid.uid58204.001', '-I', 'bond1', '-p', '192.168.56.50:3260,1', '--op=new'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:17,112-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:17,142-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2002-10.com.infortrend:raid.uid58204.112', '-I', 'bond1', '-p', '192.168.56.53:3260,1', '--op=new'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:17,174-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:17,204-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2002-10.com.infortrend:raid.uid58204.101', '-I', 'bond1', '-p', '192.168.56.52:3260,1', '--op=new'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:17,237-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:44,186-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:44,234-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:44,268-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:44,310-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:44,343-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:44,370-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:44,408-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) /var/log/vdsm/vdsm.log:2024-06-07 09:59:44,442-0400 WARN (jsonrpc/0) [storage.iscsiadm] iscsiadm executing: ['/sbin/iscsiadm', '-m', 'iface'] (iscsiadm:104) ——— The full vdsm.log is below. Thanks again, Devin
On Jun 7, 2024, at 8:14 AM, Jean-Louis Dupond <jean-louis@dupond.be> wrote:
Weird, I have the same 6.2.1.9-1 version, and here it works. You can try to add some print here: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FoVirt%2Fvdsm%2Fblob%2F4d11cae0b1b7318b282d9f90788748c0ef3cc965%2Flib%2Fvdsm%2Fstorage%2Fiscsiadm.py%23L104&data=05%7C02%7Cdevin.bougie%40cornell.edu%7C421a03a13ae14ce5822d08dc86eb74bd%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638533593041484479%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=xy9iPcpHP3jg0iyUNWlPqYBYck31Fb4bIMFzho3kFSU%3D&reserved=0<https://github.com/oVirt/vdsm/blob/4d11cae0b1b7318b282d9f90788748c0ef3cc965/lib/vdsm/storage/iscsiadm.py#L104>
This should print all executed iscsiadm commands.
On 6/06/2024 20:50, Devin A. Bougie wrote:
Awesome, thanks again. Yes, the host is fixed by just downgrading the iscsi-initiator-utils and iscsi-initiator-utils-iscsiuio packages from: 6.2.1.9-1.gita65a472.el9.x86_64 to: 6.2.1.4-3.git2a8f9d8.el9.x86_64
Any additional pointers of where to look or how to debug the iscsiadm calls would be greatly appreciated.
Many thanks! Devin
On Jun 6, 2024, at 2:04 PM, Jean-Louis Dupond <jean-louis@dupond.be> wrote:
2024-06-06 13:28:10,478-0400 ERROR (jsonrpc/5) [storage.storageServer] Could not configure connection to 192.168.56.57:3260,1 iqn.2002-10.com.infortrend:raid.sn8087428.112 and iface <IscsiInterface name='bond1' transport='tcp' netIfaceName='bond1'>: (7, b'', b'iscsiadm: Error while adding record: invalid parameter\n') (storageServer:580)
Seems like some issue with iscsiadm calls. Might want to debug which calls it does or what version change there is for iscsiadm.
"Devin A. Bougie" <devin.bougie@cornell.edu> schreef op 6 juni 2024 19:32:29 CEST: Thanks so much! Yes, that patch fixed the “out of sync network” issue. However, we’re still unable to join a fully updated 9.4 host to the cluster - now with "Failed to connect Host to Storage Servers”. Downgrading all of the updated packages fixes the issue.
Please see the attached vdsm.log and supervdsm.log from the host after updating it to EL 9.4 and then trying to activate it. Any more suggestions would be greatly appreciated.
Thanks again, Devin
On Jun 5, 2024, at 2:35 AM, Jean-Louis Dupond <jean-louis@dupond.be> wrote:
Test with that, guess it's fixed then :)
On 4/06/2024 22:33, Devin A. Bougie wrote:
Are there any known incompatibilities with RHEL 9.4 (and derivatives)?
We are running a 7-node ovirt 4.5.5-1.el8 self hosted engine cluster, with all of the hosts running AlmaLinux 9. After upgrading from 9.3 to 9.4, every node started flapping between “Up” and “NonOperational,” with VMs in turn migrating between hosts.
I believe the underlying issue (or at least the point I got stuck at) was with two of our logical networks being stuck “out of sync” on all hosts. I was unable to synchronize networks or setup the networks using the UI. A reinstall of a host succeeded but then the host immediately reverted to the same state with the same networks being out of sync.
I eventually found that if I downgraded the host from 9.4 to 9.3, it immediately became stable and back online.
Are there any known incompatibilities with RHEL 9.4 (and derivatives)? If not, I’m happy to upgrade a single node to test. Please just let me know what log files and details would be most helpful in debugging what goes wrong.
(And yes, I know we need to upgrade the hosted engine VM itself now that CentOS Stream 8 is now EOL).
Many thanks, Devin
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fprivacy-policy.html&data=05%7C02%7Cdevin.bougie%40cornell.edu%7C421a03a13ae14ce5822d08dc86eb74bd%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638533593041509929%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=9ezm7wM9wj%2B35GnoCgSA3qWbnsD%2Fdgp%2FTHtnRuTX3M0%3D&reserved=0<https://www.ovirt.org/privacy-policy.html> oVirt Code of Conduct: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt.org%2Fcommunity%2Fabout%2Fcommunity-guidelines%2F&data=05%7C02%7Cdevin.bougie%40cornell.edu%7C421a03a13ae14ce5822d08dc86eb74bd%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638533593041520348%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=eSxdcw8iYRN82Tu8X5tvrrUHVn0992lRMzuk0yX4avk%3D&reserved=0<https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovirt.org%2Farchives%2Flist%2Fusers%40ovirt.org%2Fmessage%2FPIROQCVLPVBNJY6FWLE3VSLHRAZKRB3L%2F&data=05%7C02%7Cdevin.bougie%40cornell.edu%7C421a03a13ae14ce5822d08dc86eb74bd%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638533593041529200%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=sBSnva%2Fp%2F5mkBRPOe0kBjNDQjMF%2F10%2Fw%2Bg5WdwJkN%2Bg%3D&reserved=0<https://lists.ovirt.org/archives/list/users@ovirt.org/message/PIROQCVLPVBNJY6FWLE3VSLHRAZKRB3L/>
participantes (2)
-
Devin A. Bougie -
Jean-Louis Dupond