I had a somewhat related issue. An NFS domain I was using for an iso domain failed. It set off a sequence of constantly rebooting hosts (when fencing was enabled) and constantly deactivating/reactivating hosts (when I disabled fencing). For about a day, oVirt was completely unusable. All because an iso domain died. On 2/12/19 5:16 AM, Hetz Ben Hamo wrote:
Hi,
Well, there is a severe bug that I complained about it on 4.2 (or 4.1? I don't remember) and it's regarding "yanking the power cable". Basically I'm performing a simple test: kill all hosts immediately to simulate a power loss without UPS.
For this test I have 2 nodes, and 4 storage domains: hosted_storage (that was setup during the HE installation), 1 iSCSI domain, 1 NAS domain and 1 ISO domain.
After all the nodes loose power, I power them on and the following procedure happens: 1. The node with HE finishes booting, and it takes few minutes until the HE is up. 2. When the HE is up, all the storage domains comes back to life as online and VM's with high availability starting to boot. 3. Few minutes later, *all* (with the exception of hosted_storage) storage domains are going down 4. After about 5 minutes, all the other storage domains which went down, are coming up, but by then, and VM's without high availability that are not hosted on hosted_storage remains down, you'll need to power them manually back.
This whole procedure takes about 15-25 minutes after booting the nodes, and this issue is always repeatable, just kill the power to the nodes, power them up again and see for yourself.
The solution would be to change the code and if a storage domain is up - *leave it up*, skip the check.
Thanks
On Tue, Feb 12, 2019 at 11:56 AM Sandro Bonazzola <sbonazzo@redhat.com <mailto:sbonazzo@redhat.com>> wrote:
Hi, We are planning to release the first candidate of 4.3.1 on February 20th[1] and the final release on February 26th. Please join us testing this release candidate right after it will be announced! We are going to coordinate the testing effort with a public Trello board at https://trello.com/b/5ZNJgPC3 <https://secure-web.cisco.com/19tMFHUU8AN_mu0gmv9OWNh8f-UCJz4v4zkeKzkAMiVuCUe5fdMqXHPQ9M--nvrxJDB_T1_fDOoyAXdcJmHnKH1Z9ioQyBicEiX84OtscbWZa-O0SjSLBTAeU4jOrrj9Jhc4USPWg5qIXGs8C9M78YBHjD-npSQKqWJ2J0OW2qpmNuX5vJnSawHv4c0ub6EVnUUibOr5OLGLMihoabYON37aqjY1sIVGbm_cu8tGucHvEMTf49jWDJ3_umvER6KJkIiCF7E1aWphDlHviNHftyJ4iyDqhLUB7A6o6Fjhmv9mbPkcvhi8-hh8JMzzMWe4o8y3d44L6bN1gioEVwFZWrwfgIeqzG-TEynHbmqNRb02oualCr2C7Be6j14Qk3rTvqg-mmv-7lmUWeJTeagAsVvLaeqaTmpLmPJDiv1W7jps5h7j3e5IrxRyWYBFr26pXoSo8H3ALXlYCQP7qfXvR8A/https%3A%2F%2Ftrello.com%2Fb%2F5ZNJgPC3> You'll find instructions on how to use the board there.
If you have an environment dedicated to testing, remember you can setup a few VMs and test the deployment with nested virtualization. To ease the setup of such environment you can use Lago (https://github.com/lago-project <https://secure-web.cisco.com/1BeQUAr8MMBNrI4PCobsLuO7Qe9jzr0ZvZTZGhQh384k1YRSU6NJQ_WoiLsFxvuEiIi-hlmtoimhhPmBe41sSfytf7gRUX0M7M6QId1kn8dXa9n6Ha0TO4rPyhE-_gd2vJSuEAcQpIqW7uvKs74viArMbrjeJQCEm2MTxzPBx1wblDiIbzIFWbaJ4sRYvOBHmVxiqygpFeFdnWtrSuV7k-PbW3E656dEmzSJKN5O3-vqVJm6ABC5Gyzuq9-PPdmr-I11rU7R38trx90V8zI-oyaCcWsg-ZG6hYV9eJ1YEMBaKC4Bt10_UTCiqQS0YBiQh_AvWBQfVp7mpan7BrriUBSddBRbpd4pAAoenuKOps3wTojpj75B9KFHBpmrcnescxH25eiUt2gblQv2gGiGv0FPSl0C_J8jFNmnIEkKVWOYy8QsRZ6WXfrZTtg-eGYrZHhkmCgLH65Z1Dok59ZQIYQ/https%3A%2F%2Fgithub.com%2Flago-project>)
The oVirt team will monitor the Trello board, the #ovirt IRC channel on irc.oftc.net <http://secure-web.cisco.com/1AfGdhSCI98FB8KeFrMkWEyYPf6W8Jf9hX3KNWucgkWwoyzgxT7pHASgb_tr4v0FUk1yIg0iMBHM2cWd5U9W3Oq7Xiwg97IMJ0eurn2dAkRghHPqbUI-fMLHKONlnoZxWpOu5Bw_cUt8GbKzg4H0TArA7au2Nx7mvZ8uN7KrU8MrgCr9YZAI7xzy8LGsx_7VLgkm1v1BWJWlWo0QkwMxkTjvc5U8yyQ-40I3AHrfPjAM-h_hH2aPHFKnKfLlS_QPQ5dKnCjxjWzVLP1ZDexwFB8dia9iOiEfvePf6qKLUpwK4je4hEFzPlnmvKuaXaIxSgsDM-dlLN_1DvNu3LGRtRFUdbIvzTAh_zSdwYxqRghVp0mM6b5W6wiWHq9wnSnqxDvVX0287XVF9D-nLpOvAEEHNXv9eFow-qGwz8UMetkD0Jyz6l-w8eiGW6AI0aRFCKPksn7Kag-jPJN-X64JJZaW5hBC6vyonbDXP5GK6xG4/http%3A%2F%2Firc.oftc.net> server and the users@ovirt.org <mailto:users@ovirt.org> mailing list to assist with the testing.
[1] https://www.ovirt.org/develop/release-management/releases/4.3.z/release-mana... <https://secure-web.cisco.com/1lIGHA_179QMqhSQ6BZn0ljgjmOZGmV7ZS0dusvQLWyDTssBNB4F11P-1-e7ow3YtJRivBLFqK0R7fDAwU_Ub1haKyaltZ00GzX4roujauyBvHV0nul4WBcXo-vOwfUWQK0gGPjQOHND_j7rK4lap5Z-cC0OCraVp1mVEKrz_RoKiEd3PRT7vRpM7MSEG1zTqrAxF9fCXOHyplSE-8iBByu5-G6rFC5oZ4p1Fbfru_mvQdeO7YM2ff0vd0x1oL1ygN3RD7gBmlqhEp8whpUcLpGjnpa473yQo-KsdmQs099EnDav2rjr_cxXx8-_63Nzaj7XFQwcbJptgVudvJg0E1T6mQ6C5vq3gR3-cW4V2KFJI_pv0dxcFXun3LE7RvRWXcMxS6-tag8O0f58Rt5aAxFNYuA4bzUZGg35teUfO03yRzP6xlvCAghyeIIpP_eU9uPDa-K4OpVsbAyOK38wmGw/https%3A%2F%2Fwww.ovirt.org%2Fdevelop%2Frelease-management%2Freleases%2F4.3.z%2Frelease-management.html>
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
sbonazzo@redhat.com <mailto:sbonazzo@redhat.com>
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://secure-web.cisco.com/1LXp7fKf9I-Lt1PQR-vyxchrZpgudK0tU3xkxQ3e4g01XKBJRvFS3_FNt7Joz-3-6_LBlqHSOZsxS37Ps8lPOsttuwyiwTC0-EZY2up8IsHi0ouX5ZIhApmn-MQUNW8r6CBhVbsSgc6iVMiJKJNA8xIiA1T2eZUFSmzkfJJxCwMUIGJt6_gXlaj06GILAMrEFEn6pBXObFkTrFwmHKkuSs2tlGM8aLtXlRxhMMHStBIsWfD-iFrYgUZtOPNt-ykcYX8QiBbY4w1mmHHuVbkyEOiJZfPcUjltdoo-3uknXUgNJgcS9B0sQcYT4dOwcUaAIFEq_MwrYaCN26tmEk7EHKRsL1bwPdFYI8S-eeIhuBCVNnZ21jhl2xQmQuHJijhZOT9oovjVSwhvUopkl1SbBJAQmfDAYBB4y5Yj1HhRZozJ-ERnl5nxUiY8_TjM_sos6OWFvlt8QdhxqJZdy1laQPg/https%3A%2F%2Fwww.ovirt.org%2Fsite%2Fprivacy-policy%2F> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://secure-web.cisco.com/1LFV6H1uvFJRRddPySYkUDwkxjMkxVdfh-L6rkqSL9Nub92MpmyrUYpgi72oMsYwsoey2lgVUSzxsKjsFwQb7HkBCL1Lny14wx5MvyuE6RE6EI7z2rRj0BDaikj5aCp7IGbAWFp6xtdaEoVJwFU5JWnJuGpNR-fe_cBVA5SHfo9UxgtAs9IyKEK8_hYe1tkgf3AgmUrG3d8hMu8s8xaXkazhRmueVGPN3dLSb7aJPGmay1E6mpadPcCvIRZSB9BfbTAeiK1JJizBj778Q4zRCOTZfy2uo-3hov0zsGa37vKaUaimVgU3z1QmQbV3fdUcm0hH4odfHHmz59WUbJklZYSSImAedbvMeHtjHhJc-oN9cHje6j1JKl1vl7lsb5CgXw1-HwUNI4w7vQx1EA3z98PpNyZIMW6wXEAGO3eBPXJNlah4uXv7YrPb5uGpnMC6Yn0nXRcyUWaSeuc4VEUjhkg/https%3A%2F%2Fwww.ovirt.org%2Fcommunity%2Fabout%2Fcommunity-guidelines%2F> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/URIGV3LPTE2RO2... <https://secure-web.cisco.com/1W18Wn6CeNGgBNEOQsyaVedTrcDb4_BzRek6Fd5VgtRzUzQf1VPudboj-W68iBlNZHY2XknJ1-bz-DHaDu_bijoOIVfCLwb68B_BxCoYFndg1ATyNhdBEUJYtRToekjl8CzBKhwds38HPoF4FI_MaIvQP-YiI4GKQ8Sz4QQppXhFIG9RFCKBTHE1so7s4uGaRihXYdCmP6NjWuXBssYGjSqz3Spa5X4p5JwmHmAGYmJpIb_bD8K3xoP63wvivrPYSQ45xH4MvaZV9yzzKSbi27k7ull26OChp0GKZcUCYRoH4meKitRJmeq1YfwzFpVzpj2SFDi0Fbfx8dCm7wT2x1S3nnudrskR6oxck2O0gmVqmV--R_gD1Br9-XKt78y4VpmZ0LdfikCC0_X6UHFzwdNzYOtTLqmu80zJCsR8yRdhq2SUocIRdIqCcbIqjflzevvQxNqNIbFtb1KxLmuxcTg/https%3A%2F%2Flists.ovirt.org%2Farchives%2Flist%2Fusers%40ovirt.org%2Fmessage%2FURIGV3LPTE2RO2BJFXZDHE5H5BN5I4RM%2F>
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://secure-web.cisco.com/1LXp7fKf9I-Lt1PQR-vyxchrZpgudK0tU3xkxQ3e4g01XKB... oVirt Code of Conduct: https://secure-web.cisco.com/1LFV6H1uvFJRRddPySYkUDwkxjMkxVdfh-L6rkqSL9Nub92... List Archives: https://secure-web.cisco.com/18iO89jLChpdLAQLw8j93_D8xw3sNT0sCFwecaWIN6m6nb6...