Engine trying to activate already activated hosts, causing serious instability...
I've got a 4 node cluster running oVirt 4.2.7. 3 hosts are hyperconverged with glusterfs. Engine is running on one of the gluster volumes. Since yesterday, the cluster is completely unstable and will not reliably run VMs including the Engine. A notification will pop up in the WebUI stating "Finished invoking activate on host ovirt1" while the task list indicates "Activating host ovirt1". When this task ultimately fails, the node in question goes NonOperational. All our VMs have been down since 3pm yesterday, so help is greatly appreciated. Sincerely, Jason
Hi, Please send the engine log as well as vdsm log of host ovirt1. On Wed, Nov 14, 2018 at 10:36 PM Jason P. Thomas <jthomasp@gmualumni.org> wrote:
I've got a 4 node cluster running oVirt 4.2.7. 3 hosts are hyperconverged with glusterfs. Engine is running on one of the gluster volumes. Since yesterday, the cluster is completely unstable and will not reliably run VMs including the Engine. A notification will pop up in the WebUI stating "Finished invoking activate on host ovirt1" while the task list indicates "Activating host ovirt1". When this task ultimately fails, the node in question goes NonOperational. All our VMs have been down since 3pm yesterday, so help is greatly appreciated.
Sincerely, Jason _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/DRBIRYW5THPEUC...
Thanks, Kaustav
Just figured it out...an nfs domain used for testing early on is dying/dead. Putting it into maintenance resolved the flakiness. It was still around because it housed the iso domain. Thanks for the offer to help! Jason On 11/14/18 12:53 PM, Kaustav Majumder wrote:
Hi, Please send the engine log as well as vdsm log of host ovirt1.
On Wed, Nov 14, 2018 at 10:36 PM Jason P. Thomas <jthomasp@gmualumni.org <mailto:jthomasp@gmualumni.org>> wrote:
I've got a 4 node cluster running oVirt 4.2.7. 3 hosts are hyperconverged with glusterfs. Engine is running on one of the gluster volumes. Since yesterday, the cluster is completely unstable and will not reliably run VMs including the Engine. A notification will pop up in the WebUI stating "Finished invoking activate on host ovirt1" while the task list indicates "Activating host ovirt1". When this task ultimately fails, the node in question goes NonOperational. All our VMs have been down since 3pm yesterday, so help is greatly appreciated.
Sincerely, Jason _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://secure-web.cisco.com/1VvKrU4z15mr7i4ZhpUbYB6RytKgikiGfGC6JxadpxXrMC2D9qUuJrwGjvQpYkRbZgcQB4-dfe4QgJS_gKLaGTrmuUQUQMymeTdHbbJ7MZz3IDqyI4939kU_MEN8bn5UsYAU3xmHXEfcQwp5lP6Uf44pSGMMktx7_cjSkBM8AXSiUfDOob-EGtTS9pXmGyrx7dUOVnobPh2myEBMcfOp4lRn_9ZSO6ap0AHKWGWnIohUnc2-1RYjNYDB1nt2guQdOwkt2mmirkFS6RJktKBw7a0-RPv4NlPYpLqE6Swj8qrOZ2qMsD9m6J3nycwQwwSOqKUuMO0KL0hZvWjGQpsuraLZn82Wtra_Outb8uknnC6eeaKmWhNRnFTs1zIb_Cs8Qi0R7MtwUv4YtCKAOFr-jILDQElSCpneG4vBXSiIAFpITE6IDMbLOh-48QKPHiXnC2eksz7EtsDTm4oBSwEGWfg/https%3A%2F%2Fwww.ovirt.org%2Fsite%2Fprivacy-policy%2F> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://secure-web.cisco.com/1ivH7ge8GCKkX7f09EGddrodjAiU9SknlbvYcdTGtl59RPTQVygxJKEZu6h1WL8T0BEaue6TNj1mFWXb6oh3RAT4AVUksp0u_j5J4h7zUzC97PYNhfPBZipcsghWVr9bMkD-CPLLNQmYCwpVmY4nWKFZ7GHqHVcJjIe4tj1BCeihre33DFPe18Pjo5Dot1QzYDgbr7h7s_EotYhzUa6ox9aGKgpsgYFH9UwG2ByUJb_PgRqxJbLcYxmTU7vU1KLqimWCkzgBrWFEtVoBZ_M7mZbOGu4PRft2C8jHqaXQzsfq3JH-UIZJRCHTZdM4MVwYBGCZxi3K4vXeZA5P-u1KSOQawpttEnQcSHG7exZFnAJvGA1W79VlGqgDiWZ0y7xpLCbrqZBdNtyodsTDRjw50xQG_m8i8IRHvf_UBBTFMpnmAsqjQs2yvB-pUKt7Nd3_HKwx1yxucVIdimaOoC_lZGA/https%3A%2F%2Fwww.ovirt.org%2Fcommunity%2Fabout%2Fcommunity-guidelines%2F> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/DRBIRYW5THPEUC... <https://secure-web.cisco.com/1HEEAi7LUMGP0PcqsmNbBDGs3VZNmwzipxSoKr2TT1MAyVd6V8t3Ojzgy_OaYru4AN0MfEgWViSVXX1FSE8AZOK1Yl1NgnjTH9iqRlt1K7qu_-zrPJ52iF7Vctmqz-KfdUQIcqqos8y_LIJroINsH4b8Qp_2FcqL2ewkdLHOOeDPqJ3P1Ob3Q-CwND8BYjl2KOpJloxTRHyBeCRNK2ttWhV6likSFH_qz1ld44dqbCNjZuwRNyHuWSqWMIUs3-ug-PwbAL_Kk-pAzOcsIVEchtAn7vcM812efZoKLxDCk1gvoVD8gpRi6rQMr8vCUWuwlJ9L8jIY5buor1QzW26DWdI1q8PqI39A5ztGkcXgoFRuflRvcQSZuRANTytaLWTvNSxO-2py8-DmuYO3bGAP_OH23vanZ7BgpyyR_POv6_R_xnwwrdCRtrOZEwl41IIjHeH5Scl7YOLSSfiVSm5qW9A/https%3A%2F%2Flists.ovirt.org%2Farchives%2Flist%2Fusers%40ovirt.org%2Fmessage%2FDRBIRYW5THPEUCNBEQNYLD5WNSBBCKS5%2F>
Thanks, Kaustav
No problem On Wed, Nov 14, 2018 at 11:33 PM Jason P. Thomas <jthomasp@gmualumni.org> wrote:
Just figured it out...an nfs domain used for testing early on is dying/dead. Putting it into maintenance resolved the flakiness. It was still around because it housed the iso domain. Thanks for the offer to help!
Jason
On 11/14/18 12:53 PM, Kaustav Majumder wrote:
Hi, Please send the engine log as well as vdsm log of host ovirt1.
On Wed, Nov 14, 2018 at 10:36 PM Jason P. Thomas <jthomasp@gmualumni.org> wrote:
I've got a 4 node cluster running oVirt 4.2.7. 3 hosts are hyperconverged with glusterfs. Engine is running on one of the gluster volumes. Since yesterday, the cluster is completely unstable and will not reliably run VMs including the Engine. A notification will pop up in the WebUI stating "Finished invoking activate on host ovirt1" while the task list indicates "Activating host ovirt1". When this task ultimately fails, the node in question goes NonOperational. All our VMs have been down since 3pm yesterday, so help is greatly appreciated.
Sincerely, Jason _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://secure-web.cisco.com/1VvKrU4z15mr7i4ZhpUbYB6RytKgikiGfGC6JxadpxXrMC2D9qUuJrwGjvQpYkRbZgcQB4-dfe4QgJS_gKLaGTrmuUQUQMymeTdHbbJ7MZz3IDqyI4939kU_MEN8bn5UsYAU3xmHXEfcQwp5lP6Uf44pSGMMktx7_cjSkBM8AXSiUfDOob-EGtTS9pXmGyrx7dUOVnobPh2myEBMcfOp4lRn_9ZSO6ap0AHKWGWnIohUnc2-1RYjNYDB1nt2guQdOwkt2mmirkFS6RJktKBw7a0-RPv4NlPYpLqE6Swj8qrOZ2qMsD9m6J3nycwQwwSOqKUuMO0KL0hZvWjGQpsuraLZn82Wtra_Outb8uknnC6eeaKmWhNRnFTs1zIb_Cs8Qi0R7MtwUv4YtCKAOFr-jILDQElSCpneG4vBXSiIAFpITE6IDMbLOh-48QKPHiXnC2eksz7EtsDTm4oBSwEGWfg/https%3A%2F%2Fwww.ovirt.org%2Fsite%2Fprivacy-policy%2F> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://secure-web.cisco.com/1ivH7ge8GCKkX7f09EGddrodjAiU9SknlbvYcdTGtl59RPTQVygxJKEZu6h1WL8T0BEaue6TNj1mFWXb6oh3RAT4AVUksp0u_j5J4h7zUzC97PYNhfPBZipcsghWVr9bMkD-CPLLNQmYCwpVmY4nWKFZ7GHqHVcJjIe4tj1BCeihre33DFPe18Pjo5Dot1QzYDgbr7h7s_EotYhzUa6ox9aGKgpsgYFH9UwG2ByUJb_PgRqxJbLcYxmTU7vU1KLqimWCkzgBrWFEtVoBZ_M7mZbOGu4PRft2C8jHqaXQzsfq3JH-UIZJRCHTZdM4MVwYBGCZxi3K4vXeZA5P-u1KSOQawpttEnQcSHG7exZFnAJvGA1W79VlGqgDiWZ0y7xpLCbrqZBdNtyodsTDRjw50xQG_m8i8IRHvf_UBBTFMpnmAsqjQs2yvB-pUKt7Nd3_HKwx1yxucVIdimaOoC_lZGA/https%3A%2F%2Fwww.ovirt.org%2Fcommunity%2Fabout%2Fcommunity-guidelines%2F> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/DRBIRYW5THPEUC... <https://secure-web.cisco.com/1HEEAi7LUMGP0PcqsmNbBDGs3VZNmwzipxSoKr2TT1MAyVd6V8t3Ojzgy_OaYru4AN0MfEgWViSVXX1FSE8AZOK1Yl1NgnjTH9iqRlt1K7qu_-zrPJ52iF7Vctmqz-KfdUQIcqqos8y_LIJroINsH4b8Qp_2FcqL2ewkdLHOOeDPqJ3P1Ob3Q-CwND8BYjl2KOpJloxTRHyBeCRNK2ttWhV6likSFH_qz1ld44dqbCNjZuwRNyHuWSqWMIUs3-ug-PwbAL_Kk-pAzOcsIVEchtAn7vcM812efZoKLxDCk1gvoVD8gpRi6rQMr8vCUWuwlJ9L8jIY5buor1QzW26DWdI1q8PqI39A5ztGkcXgoFRuflRvcQSZuRANTytaLWTvNSxO-2py8-DmuYO3bGAP_OH23vanZ7BgpyyR_POv6_R_xnwwrdCRtrOZEwl41IIjHeH5Scl7YOLSSfiVSm5qW9A/https%3A%2F%2Flists.ovirt.org%2Farchives%2Flist%2Fusers%40ovirt.org%2Fmessage%2FDRBIRYW5THPEUCNBEQNYLD5WNSBBCKS5%2F>
Thanks, Kaustav
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VZL4FJOSBQH2QN...
Thanks, Kaustav
participants (2)
-
Jason P. Thomas -
Kaustav Majumder